Ever wondered what your PHP application is doing under the hood? Want to understand the system calls it makes? This blog provides practical examples to follow along with the blog post and explore the fascinating world of system call tracing in PHP applications.
You can find the Github repo to reproduce the scripts of this blog here.
“strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state.”
AKA, a tool you do not want to use. Is hard to read. But, it _can_ help. Sometimes.
Truth be told, in the perfect world you have Sentry implemented along with trace id’s across events and API requests. Sadly, this is not always implemented, making your life way harder.
Table of Contents
When do you Strace PHP
So, let’s assume a project that has some weird kind of magic. A project that you never want to touch or understand. And you get the report “hey there is a blank page, go fix”. What do you do?
At first, you will think Sentry or something similar. Only to find out, that Sentry did not catch it. Perhaps it was not installed, initiated in time or the best you’ll get is a server 500 error. So, what do you do? Enable more debugging in PHP (if possible). But, this does not always works. So what’s next?
When debugging a PHP application, selecting the right tool often depends on the layer at which the problem exists and whether you’re working on a development machine or a live production system.
Strace
What It Does:
Traces system calls and signals to reveal how your PHP process interacts with the Linux kernel. It’s great for pinpointing issues like file access errors, network timeouts, or missing libraries.
Live System Suitability:
You can use strace on a live system, but caution is advised. Its output can be voluminous, and the tool itself may impact performance slightly if used for extended periods. It’s best reserved for targeted diagnostics rather than continuous monitoring.
Ideal Scenario:
When you suspect the root cause is at the OS level—for example, if your PHP script is failing to open files or experiencing unexpected network delays—strace gives you the detailed view needed to diagnose such issues.
Xdebug
What It Does:
Focuses on the internal workings of PHP. With Xdebug, you can step through code, inspect variables, and generate stack traces, making it a powerful tool for debugging code logic and performance within the PHP runtime.
Live System Suitability:
Xdebug is typically used in development environments. Running it on a live system is not recommended because it introduces overhead and may expose sensitive details about your code.
Ideal Scenario:
When you need to understand the flow of your PHP code—setting breakpoints and examining variable values—Xdebug is your go-to tool. It provides a much clearer picture of your application logic compared to the low-level details offered by strace.
APM Tools (e.g., New Relic)
Application Performance Monitoring (APM) tools like New Relic offer a broad overview of your application’s performance. They track metrics such as response times, error rates, and transaction traces, which can help identify bottlenecks or anomalies in production.
Live System Suitability:
APM tools are designed to run continuously in live environments. They’re optimized to have a minimal impact on performance while providing real-time insights into your application’s behavior.
Ideal Scenario:
If you’re looking for a high-level view to monitor performance and catch issues before they escalate, an APM tool is ideal. It’s less about digging into the minutiae of system calls or PHP internals and more about ensuring that your application runs smoothly under real-world conditions.
Use strace when you need to diagnose low-level OS interactions, keeping in mind its careful use on live systems due to potential performance impacts. I’ll go into how to do this using this blog.
Choose Xdebug for in-depth PHP code debugging in a controlled development setting.
Opt for an APM tool like New Relic when you need continuous, real-time performance monitoring on a live production system.
How to use it?
Tl;dr strace php file.php
The most useful feature is the -s flag, aka string limit. It allows for a more useful output. The more you allow, the more performance (and time) it requires to get said output. `strace -s 100`But you will find it useful, since 32 characters with PHP is a bit short.
For more information, check out man strace.
Simulating and Inspecting Your Environment Settings
Before you run your PHP scripts with strace, it’s a good idea to know what settings your application is using. This includes configuration from php.ini, environment variables, and settings defined in your deployment tools (such as Docker).
If you want to quickly check which environment variables are set, you can create a temporary web endpoint. This endpoint can be used in a controlled environment without impacting production. For example, you might create a file called show-env.php with the following content:
<?php
print_r(getenv());
?>
Deploy this file on a non-production instance or behind an access restriction so that it isn’t publicly accessible. This will output all environment variables available to your PHP process.
Environment variables can be defined in various ways:
- System Settings: Directly in the operating system.
- /etc/environment
- user profile
- etc
- Configuration Files: Like .env files or server configuration.
- Nginx or Apache can pass environment settings.
- Docker Containers: Passed as environment variables in your Docker Compose or Docker run commands.
Strace PHP examples
Where is this error coming from?
So, we have a file which results in a blank page. Let’s run the file using strace.
docker compose run --rm app strace php HiddenError.php
This will produce the following output:
execve("/usr/local/bin/php", ["php", "HiddenError.php"], 0xffffc204d018 /* 15 vars */) = 0
brk(NULL) = 0xaaab26a0c000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffb4e6b000
faccessat(AT_FDCWD, "/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=8862, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8862, PROT_READ, MAP_PRIVATE, 3, 0) = 0xffffb4e68000
close(3) = 0
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libreadline.so.8", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
[..]
read(3, "28968957 /usr/"..., 1024) = 1024
read(3, "/aarch64-linux-gnu/libcrypto.so."..., 1024) = 1024
read(3, " 00196000 00:35 28978730 "..., 1024) = 1024
read(3, "ch64-linux-gnu/libreadline.so.8."..., 1024) = 1024
read(3, "030000 00:35 28968841 "..., 1024) = 170
close(3) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
getpid() = 9
getpid() = 9
openat(AT_FDCWD, "HiddenError.php", O_RDONLY) = 3
getcwd("/app", 4096) = 5
newfstatat(AT_FDCWD, "/app/HiddenError.php", {st_mode=S_IFREG|0644, st_size=50, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/app", {st_mode=S_IFDIR|0755, st_size=256, ...}, AT_SYMLINK_NOFOLLOW) = 0
rt_sigaction(SIGPROF, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGPROF, {sa_handler=0xaaaae95a2ec0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGHUP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGHUP, {sa_handler=0xaaaae95a2ec0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0xaaaae95a2ec0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=0xaaaae95a2ec0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=0xaaaae95a2ec0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGUSR1, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGUSR1, {sa_handler=0xaaaae95a2ec0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGUSR2, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGUSR2, {sa_handler=0xaaaae95a2ec0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGPROF, {sa_handler=0xaaaae95a2ec0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_ONSTACK|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
newfstatat(0, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0
newfstatat(0, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0
newfstatat(1, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0
newfstatat(1, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0
newfstatat(2, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0
newfstatat(2, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0
getcwd("/app", 4096) = 5
ioctl(3, TCGETS, 0xfffff928d010) = -1 ENOTTY (Inappropriate ioctl for device)
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=50, ...}, AT_EMPTY_PATH) = 0
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=50, ...}, AT_EMPTY_PATH) = 0
read(3, "<?php\ninclude('includes/HiddenEr"..., 4096) = 50
getcwd("/app", 4096) = 5
newfstatat(AT_FDCWD, "/app/./includes/HiddenErrorInclude.php", {st_mode=S_IFREG|0644, st_size=114, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/app/./includes", {st_mode=S_IFDIR|0755, st_size=96, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/app/includes/HiddenErrorInclude.php", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=114, ...}, AT_EMPTY_PATH) = 0
read(4, "<?php\nerror_reporting(0);\nini_se"..., 114) = 114
close(4) = 0
close(3) = 0
getpid() = 9
munmap(0xffffb0d27000, 196608) = 0
munmap(0xffffb1020000, 197952) = 0
munmap(0xffffb0db0000, 299504) = 0
munmap(0xffffb0d80000, 131104) = 0
munmap(0xffffb0e00000, 2097152) = 0
munmap(0xffffb0cd6000, 331776) = 0
futex(0xffffb4a8218c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
exit_group(255) = ?
+++ exited with 255 +++
Did you find the error? No?
Let’s take it apart. At the near end, you will find this:
openat(AT_FDCWD, "/app/includes/HiddenErrorInclude.php", O_RDONLY) = 4
Ah, we have to be in includes/HiddenErrorInclude.php. Mind you, /app is the volume mount in Docker. When opening this file, we can find that a non-existing method is being used.
We can increase the output, but in this case this won’t help. Why? because the error is still hidden in PHP. But at least we know in which file we have to be and over there, we can enable displaying PHP errors or something. But, at least it is more than a blank page.
docker compose run --rm app strace -s 1000 php HiddenError.php
Strace to debug long loading times
A long loading time usually indicates a firewall issue. Usually, the port you want to connect to is blocked. However, it could be caused by other factors as well. Let’s say a connection to a remote service or a service (e.g. database) requires a long time to produce a response. Either way, they can be tricky to debug when tools such as Sentry are either not working or absent.
When your application connects to remote services, the PHP process has usually a CPU load of around 0%. Mostly, it just idles and waits for the remote service to respond. You can check this by opening top on the server while making the PHP request or running the command. In such cases, you may see situations where a number of PHP processes are running but they are idling.
To debug, you would run strace and check for any connectivity output.
strace php LongLoadingTime.php
I won’t include the entire strace output here since it is a bit much (1000+ rows), but it may overload your terminal with too much data. Reading the strace manual, shows that you can use the following command instead to debug network connectivity:
strace -f -e trace=network php LongLoadingTime.php
What does this do? So, we have supplier the parameters – f -e trace=network. When we look at the man page, we can find the following:
-f
--follow-forks
Trace child processes as they are created by currently
traced processes as a result of the fork(2), vfork(2) and
clone(2) system calls. Note that -p PID -f will attach all
threads of process PID if it is multi-threaded, not only
thread with thread_id = PID.-e trace=syscall_set
-e t=syscall_set
--trace=syscall_set
Trace only the specified set of system calls. syscall_set
is defined as [!]value[,value], and value can be one of the
following:
[..]
Any forks created by your process is also followed by strace, making it more complete. And the -e trace=network argument allows you to filter the output on the network. This prevents that your terminal crashes due to the humongous output.
So what is the output? Let’s see by running the strace.
application@f226da0aa738:/app$ strace -f -e trace=network php LongLoadingTime.php
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 4
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 4
setsockopt(4, SOL_IP, IP_RECVERR, [1], 4) = 0
connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.11")}, 16) = 0
sendmmsg(4, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="v\272\1\0\0\1\0\0\0\0\0\0\10httpstat\2us\0\0\1\0\1", iov_len=29}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=29}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="3\270\1\0\0\1\0\0\0\0\0\0\10httpstat\2us\0\0\34\0\1", iov_len=29}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=29}], 2, MSG_NOSIGNAL) = 2
recvfrom(4, "v\272\201\200\0\1\0\1\0\0\0\0\10httpstat\2us\0\0\1\0\1\300\f\0"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.11")}, [28 => 16]) = 45
recvfrom(4, "3\270\201\200\0\1\0\0\0\0\0\0\10httpstat\2us\0\0\34\0\1", 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.11")}, [28 => 16]) = 29
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("20.40.202.3")}, 16) = -1 EINPROGRESS (Operation now in progress)
getsockopt(4, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
In this case, connect() and inet_addr() indicates an attempt to connect to a remote service is being made. Until a timeout is reached, the strace will remain at this point until the timeout is reached. It allows you to read out at which point it is stuck.
Based on the sendmmsg, you can deduct that I am connecting to httpstat.us. On line 10, you can see that it connects to a DNS service (runs on port 53) to resolve the name. However, it’s possible that you won’t see something like this. In such cases, the PHP script may connect directly to an IP.
The IP is shown on line 15, with inet_addr. The port is shown in htons(443), meaning https. While the strace is stuck on this row, PHP is still running and waiting for a completed request. You can search your environment settings and code for the usage of the IP. If you cannot find this, you may want to remove the filter and dig (starting from the connect line and upwards) for any references of file inclusions to further debug the culprit.
Issues can be an incorrectly configured setting or there are network errors to connect to that remote IP. Using the Linux command line nc can show you if a connection can be made:
nc -vz 20.40.202.3 443
Connection to 20.40.202.3 port 443 [tcp/https] succeeded!
# or a failed example:
nc -vz 127.0.0.1 444
nc: connectx to 127.0.0.1 port 444 (tcp) failed: Connection refused
I have left out the remaining part of the strace output. In this case, there is a clear error, but when you are running strace this is usally not the case.
getsockopt(4, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
setsockopt(4, SOL_TCP, TCP_ULP, [7564404], 4) = -1 ENOENT (No such file or directory)
Connection timed out after 60.45 seconds
Error: file_get_contents(https://httpstat.us/200?sleep=120000): Failed to open stream: HTTP request failed!
+++ exited with 0 +++
application@f226da0aa738:/app$
No CLI, but http only
PHP has a handy feature: a build-in PHP webserver. Just enter the directory you are in and hit `php -S 0.0.0.0:8000`, which opens on all ports port 8000 for your webserver. This is useful in case the error can only be reproduced using a webserver. While it is possible to strace apache with mod_php or php-fpm, it is usually way more complicated and affects production. The benefit of running PHP while the webserver remains running, is that you can usually do this on a production server as well. Please note: in no way I recommend doing this. There is always a risk. Try to replicate the error first on local, testing or acceptance environments first.
Here is an example. HttpOnlyLongLoadingTime.php. Let’s run this script:
# php HttpOnlyLongLoadingTime.php
This script can only be accessed via HTTP
Hmmm, how should we debug this? Using strace with a webserver is challenging. So, let’s start php as a webserver. With strace.
# Let's start the webserver the filter turned on
docker compose run --rm app strace -f -e trace=network php -S 127.0.0.1:8000
# In a separate terminal, we will make a request:
docker compose exec app curl http://localhost:8000/HttpOnlyLongLoadingTime.php
When we open the same PHP script using http, you will see that we have some output and can debug the application. Since we are using curl, we can simulate almost any http request. If you are really up for it, you could try to simulate a browser request.
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
listen(3, 4096) = 0
[Wed Feb 26 20:39:57 2025] PHP 8.4.4 Development Server (http://127.0.0.1:8000) started
accept(3, {sa_family=AF_INET, sin_port=htons(37670), sin_addr=inet_addr("127.0.0.1")}, [16]) = 4
[Wed Feb 26 20:40:02 2025] 127.0.0.1:37670 Accepted
recvfrom(4, "GET /HttpOnlyLongLoadingTime.php"..., 16383, 0, NULL, NULL) = 105
sendto(4, "HTTP/1.1 200 OK\r\nHost: localhost"..., 162, 0, NULL, 0) = 162
sendto(4, "Testing connection to https://ht"..., 59, 0, NULL, 0) = 59
sendto(4, "Timeout set to: 5 seconds\n", 26, 0, NULL, 0) = 26
sendto(4, "Remote sleep set to: 120 seconds"..., 34, 0, NULL, 0) = 34
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 6
connect(6, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 6
connect(6, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 6
setsockopt(6, SOL_IP, IP_RECVERR, [1], 4) = 0
connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.11")}, 16) = 0
sendmmsg(6, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\30\36\1\0\0\1\0\0\0\0\0\0\10httpstat\2us\0\0\1\0\1", iov_len=29}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=29}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\374\37\1\0\0\1\0\0\0\0\0\0\10httpstat\2us\0\0\34\0\1", iov_len=29}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=29}], 2, MSG_NOSIGNAL) = 2
recvfrom(6, "\30\36\201\200\0\1\0\1\0\0\0\0\10httpstat\2us\0\0\1\0\1\300\f\0"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.11")}, [28 => 16]) = 45
recvfrom(6, "\374\37\201\200\0\1\0\0\0\0\0\0\10httpstat\2us\0\0\34\0\1", 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.11")}, [28 => 16]) = 29
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 6
connect(6, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("20.40.202.3")}, 16) = -1 EINPROGRESS (Operation now in progress)
getsockopt(6, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
setsockopt(6, SOL_TCP, TCP_ULP, [7564404], 4) = -1 ENOENT (No such file or directory)
[Wed Feb 26 20:40:07 2025] 127.0.0.1:37670 [200]: GET /HttpOnlyLongLoadingTime.php
[Wed Feb 26 20:40:07 2025] 127.0.0.1:37670 Closing
shutdown(4, SHUT_RDWR) = 0
No CLI, http only with browser requirement
The only thing that is useful in this case, is that you can create an SSH tunnel to the remote server. Update the local /etc/hosts and you can use the remote server (probably) to simulate the error. Using curl/wget locally would be more optimal, but may be in some cases more challenging.
You could tunnel the remote 8000 port and map it to your local port 80. You could also update your /etc/hosts and point the domain you are trying to reach, to the local application. This may all help to debug the application. There are some more advanced situations (e.g. the code forces https) before you can debug the code.
You can find more information about tunneling here: https://www.ssh.com/academy/ssh/tunneling-example
FAQ
I have certainly not covered all features and debuggin of strace in this blog, but hopefully this helps you out with debugging. I have listed some questions or topics that might be interesting for you.
When the output from strace becomes overwhelming, consider these approaches:
Limit the Scope: Use flags like -e trace=network or -e trace=file to narrow down the output to a specific type of system call.
Increase the String Limit Judiciously: While the -s flag can provide more detail, balance the need for information against performance impact.
Filter by Process or Thread: If your application spawns multiple processes or threads, use options to focus on a particular PID, which can help isolate the problem area.
Debugging multi-threaded applications can be challenging because strace follows each thread’s system calls, resulting in interleaved output. Here are a few strategies:
Use the -f Option: This flag allows you to follow forked processes or threads, but be prepared for mixed output.
Focus on Specific Threads: If possible, identify the thread that’s most likely responsible for the issue and filter the output for that thread’s PID.
Post-Processing Tools: Consider using scripts or log analysis tools to parse and separate the output based on thread IDs. This can make it easier to follow the flow of individual threads.
By addressing these common pitfalls, you can fine-tune your debugging process and avoid getting lost in a sea of data. This section should help readers navigate the complexities of using strace in real-world scenarios, making it easier to extract actionable insights from the output.
Yes, but with Caution: While strace can be run on live systems for short periods, prolonged tracing may impact performance.
Short, Targeted Traces: Limit the duration and scope of your trace to minimize overhead.
Use in a Staging Environment: Whenever possible, replicate the issue in a non-production environment to conduct a more thorough investigation.
Start with narrow filters (e.g., a specific syscall or process) rather than a full system trace. You will notice performance degradation using strace. Therefore, do not use it while benchmarking unless this helps debugging. Real world scenarios will show different performance numbers.
Monitor system performance concurrently to ensure that the tracing does not introduce further issues.
Combine with Application-Level Debugging: Tools like Xdebug can complement strace by providing insights at the code level.
Use with APM Tools: If you suspect that system-level delays are affecting overall performance, pair strace findings with data from APM solutions for a holistic view.
Yes, tools like ltrace (to trace library calls) or systemtap can offer different angles of insight.
For containerized environments, tools like Sysdig or eBPF-based solutions might offer more granularity without affecting the container’s performance as heavily.
If your strace output suggests stalled network calls (e.g., persistent EINPROGRESS errors), cross-check with your firewall settings or DNS configuration.
If strace shows frequent file-not-found or permission errors, verify the file system paths and user permissions. Sometimes, the issue might be as simple as a misconfigured file mount in a Docker environment. Or it could be a corrupted file system.
When your PHP application interacts with remote services, strace can reveal delays during connection attempts. Use this information to check network latency or misconfigurations in the service endpoints.