1.6G /root/strace.sec_adv file.

**stj** · 08-11-2016, 06:25 PM

Re: 1.6G /root/strace.sec_adv file.

could have been a hack attempt - someone tried to envoke GDB

you dont do that after something fails, you have to envoke it first and then call the suspect executeable.

so what you have here, is someone or something tried to envoke a debug enviroment - possibly to try to hack around or through something.

the good news is the files where not present or atleast in the expected place.
unless the rest of the log is attempts to execute the same file from hundreds of different paths!

**Spork Schivago** · 08-11-2016, 06:56 PM

Re: 1.6G /root/strace.sec_adv file.

Originally posted by stj

could have been a hack attempt - someone tried to envoke GDB

you dont do that after something fails, you have to envoke it first and then call the suspect executeable.

so what you have here, is someone or something tried to envoke a debug enviroment - possibly to try to hack around or through something.

the good news is the files where not present or atleast in the expected place.
unless the rest of the log is attempts to execute the same file from hundreds of different paths!

I looked a bit further into the file, I do not think it was a hacking attempt Stj. The libgdbm library did get loaded. It seems whatever program was running tried various places, looking for the library, before finding it at /usr/lib64/libgdbm.so.2

I see a bunch of different calls like that. Then, after the file / library is successfully found, the strace file shows the contents of the file. It's displayed like this:

Code:

10803 read(3, "\177ELF\2\1\1\0\0\0....

After the libraries get loaded, I see /usr/local/cpanel/scripts/check_security_advice_changes got read.

This is a script that should be executed fairly often, to check basic security things (for example, is symlink protection enabled, is clamav installed and enabled, etc).

I know that script had issues and wasn't working properly. cPanel was aware of it and recently released a supposed fix for it. Updates generally don't come often, but we've had quite a few cPanel updates lately. One was released and the very next day, another one released. I'm guessing one of the updates might have broken something and perhaps they fixed it the next day.

There's been 7 updates to cPanel since July 25th (the day the file was created). The first update after the file was created was July 26th.

The changelog shows what they fixed and for the release on July 26th, one is sticking out to me:

Code:

Implemented case CPANEL-7613: Temporary Tweak Setting to enable security advisor notices.

So, I know for certain cPanel knew there was something wrong with that script. I reported it myself to them and they said they were already aware of it and expecting a fix. I'm guessing perhaps their fix didn't work as expected? It seems on 08/08/2016, they released another fix for that script.

I wish I could open the file to view near the end. I tail it but there's not enough information displayed. I try opening it with nano and nano just dies because of the file size and the limited amount of RAM I have. Downloading the file locally will take a very long time.

**Spork Schivago** · 08-11-2016, 07:01 PM

Re: 1.6G /root/strace.sec_adv file.

I'm thinking maybe the sec_adv extension might be referring to security advisor...The strace is just filled with the check_security_advice_changes and every library or script or executable that got called from check_security_advice_changes it looks like. I see one error a lot here:

Code:

10803 ioctl(9, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fffaa07fcf0) = -1 ENOTTY (Inappropriate ioctl for device)

Perhaps this is why the strace file was created? I've reached out to cPanel. Perhaps they can provide a little more insight. I don't ever remember the Linux kernel creating strace files when a program crashes or anything. Maybe they got some weird setting that creates these files if something crashes?

**Spork Schivago** · 08-11-2016, 07:10 PM

Re: 1.6G /root/strace.sec_adv file.

July 26th was when I realized modsecurity2 was no longer installed or enabled. To this date, cPanel wasn't able to figure out how it got uninstalled. They escalated my ticket, because if this was a bug, it could have very huge implications. In the end, they said from the best of their knowledge, my server wasn't compromised. I don't think it was but maybe this is related. Before that, I made the switch from EasyApache3 to EasyApache4 and there were issues. Things didn't go the way they should have. Maybe this file was created during the EasyApache3 to EasyApache4 migration.

**stj** · 08-11-2016, 07:23 PM

Re: 1.6G /root/strace.sec_adv file.

i find it strange and very wrong that stuff like GDB is even installed on a webserver.

it's only needed by developers.
i hope that does not mean GCC is also present!

**Spork Schivago** · 08-11-2016, 08:10 PM

Re: 1.6G /root/strace.sec_adv file.

Originally posted by stj

i find it strange and very wrong that stuff like GDB is even installed on a webserver.

it's only needed by developers.
i hope that does not mean GCC is also present!

Although there is a webserver running on my server, it's mainly used for development purposes at this point in time. I have gcc installed but disabled for everyone besides root. Only root can execute gcc at this point in time. I see gdb is from the base repository. Perhaps it comes by default? I'm guessing someone compiled stuff with debugging information in tack and never stripped the libraries / files. This allows debugging to be possible.

There are things that do worry me a bit. chkrootkit displays hidden files in the /proc directory. They don't start with a .

I don't know how this is done without modifying the file system. Maybe it's something to do with the virtual environment? chkrootkit believes it's lkm. rkhunter doesn't think so though.

Here's an example:

Code:

./chkproc -v
PID  706(/proc/706): not in readdir output
PID  706: not in ps output

ls -ld /proc/706*
/bin/ls: cannot access /proc/706*: No such file or directory

ls -ld /proc/706
dr-xr-xr-x 7 root root 0 Aug 11 21:51 /proc/706/

ps aux | grep 706
root  1555 0.0 0.0 103308  868  pts/0  S+  21:58  0:00 grep --color=yes 706

cat /proc/706/cmdline
/sbin/rsylogd-i/var/run/syslogd.pid-c5

ps aux|grep -i rsyslogd
root [B]705[/B] 0.0 0.0  249084  1468 ?   Sl  21:50  0:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5

cat /proc/705/cmdline
/sbin/rsyslogd-i/var/run/syslogd.pid-c5

stat /proc/706
 File: `/proc/706'
 Size: 0        Blocks: 0     IO Block: 1024  directory
Device: 4ch/76d Inode: 1902207342 Links: 7
Access: (0555/dr-xr-xr-x)  Uid: (  0/  root)   Gid: (   0/   root)
Access: 2016-08-11 21:51:22.029777867 -0400
Modify: 2016-08-11 21:51:22.029777867 -0400
Change: 2016-08-11 21:51:22.029777867 -0400

I just restarted the server before posting this, hence the new file times.

There's a whole bunch of these hidden proc entries. They never showed up until last night, to my knowledge. chkproc never showed anything until it e-mailed me last night. They're all either rsyslogd, mysql (MariaDB), named, or nscd.

Soooo, rkhunter doesn't detect anything but chkrootkit does. We all know chkrootkit doesn't do a very good job of detecting lkm and rkhunter does a much better job, so I think we can assume these are false positives. But why can't I see them under the /proc directory, yet I can access them if I provide a full path name?

How much do you know about Linux and programming Stj? I'm thinking perhaps the main binary (let's say rsyslogd for instance) executes and then it spawns. When it spawns, does the child process show up visible under the /proc directory or no? Rebooting didn't fix the issue.

**Spork Schivago** · 08-11-2016, 08:22 PM

Re: 1.6G /root/strace.sec_adv file.

I forgot /proc wasn't a real filesystem, but rather a virtual filesystem that exists in memory. I'm wondering what changed that created these files. Last thing I remember doing to the server was enabling the named caching service...there's been a few updates the past few days, maybe one of them changed something.

**Spork Schivago** · 08-11-2016, 09:23 PM

Re: 1.6G /root/strace.sec_adv file.

I see in the MariaDB database's myhostname.mydomain.com.err file,

Code:

[Note] /usr/sbin/mysql (mysqld 10.1.16-MariaDB) starting as process 951 ...
[Note] InnoDB: Using mutexes to ref count buffer pool pages

I google mutexes and see:

Code:

In computer programming, a mutex (mutual exclusion object) is a program object that is created so that multiple program thread can take turns sharing the same resource, such as access to a file.

I'm thinking this explains the "hidden" directories in the /proc directory. I think they're just mutexes that are created for multi-core CPUs. What do you guys think? I'd like to think if something bad was going on, those hidden directories cmdline files would show something besides mysql, nscd, named and rsyslogd running. MariaDB is still listening just on the local loopback. As far as I can tell, no one has gotten in. No new usernames, password authentication for ssh is disabled, the Apache webserver's page hasn't changed, nothing new in the docroot directory, no suspicious programs running. I think I'm good.

**stj** · 08-11-2016, 11:03 PM

Re: 1.6G /root/strace.sec_adv file.

you could try to procdump 706 as an elf binary - then look at it with a hex editor.

Attached Files

skpd.tar.zip (5.1 KB, 4 views)

**Spork Schivago** · 08-12-2016, 11:24 AM

Re: 1.6G /root/strace.sec_adv file.

Stj,

I cloned the github repository for Skpd and it didn't work. All the auxv are out of range. Quickly looking at the source code, I see a comment right before the check. It says // check that it is a valid PHDR.

I'm guessing because these processes don't show up under programs like top or ps aux, chances are good programs like skpd won't work. I've contacted MariaDB and I've asked them to see if they know why their database engine might be creating hidden directories under the /proc filesystem. They called back when I wasn't here and left a message saying to call them back. I'm getting ready to do that.

I'll keep ya posted.

**stj** · 08-12-2016, 12:07 PM

Re: 1.6G /root/strace.sec_adv file.

if skpd cant catch it then it's shadowing or detaching itself to avoid strace.
this is only done for software protection - either by company's or by hackers trying to hide themselves.

**Spork Schivago** · 08-12-2016, 12:08 PM

Re: 1.6G /root/strace.sec_adv file.

Okay, I've just got off the phone with the MariaDB person who called. Unfortunately, he was just a sales person but he said he'll pass my question along to one of the technical support people and they will call me back on Monday.

So, I created a very simple C program that creates a few threads and "pauses" waiting for me to hit enter. I have it setup to create two threads and then use a getchar() waiting for user input, after printing a message to the screen.

I run ./chkproc -v again, but I'm now using a modified version. My modified version doesn't just show the hidden procs, but it also displays the contents of /proc/<hidden_proc>/cmdline.

And guess what? Sure enough, my threads program is listed twice as a hidden process.

Code:

./threads
Hello from thr_func, thread id: 0
Hello from thr_func, thread id: 1
<waiting for me to hit enter>

<from another shell>
ps aux |grep -i thread
USER   PID %CPU %MEM   VSZ  RSS TTY   STAT START  TIME COMMAND
root    2  0.0  0.0    0   0 ?    S  Aug11  0:00 [kthreadd/79271]
root  21861  0.0  0.0  26716  588 pts/1  Sl+ 13:51  0:00 ./threads
root  21962  0.0  0.0  103312  1004 pts/0  S+  13:53  0:00 grep --color=yes -i thread

Then, I run my modified chkproc version:

Code:

./mychkproc -v
(lots of hidden processes)
(last two entries)
PID 21862 (/proc/21862): not in readdir output
PID 21862: not in ps output
cat /proc/21862/cmdline
./threads

PID 21863(/proc/21863): not in readdir output
PID 21863: not in ps output
cat /proc/21863/cmdline
./threads

You have   75 process hidden for readdir command
You have   75 process hidden for ps command

I wonder if there's away by reading something in the /proc/<pid> directory to tell if the program is a thread, so I can modify chkproc's source code to ignore threaded processes. It'd be nice to be able to find the parent PID as well, so I can display a message saying hey, we found a few threads, these are the parent PIDs for them.

**Spork Schivago** · 08-12-2016, 12:40 PM

Re: 1.6G /root/strace.sec_adv file.

I think the /proc/<pid>/stat file gives me the information I need. Here's a snippet of the file:

Code:

cat /proc/21862/stat
21862 (threads) S 5529 21861 5498 34817 21861....

It looks like the second entry in that file tells me the program name and the 5th and 8th entries in that file tells me the parent process...

If I look at the parent processed /proc directory, there's a task sub-directory. If I go in there, I see the parent AND child PIDs.

1.6G /root/strace.sec_adv file.

1.6G /root/strace.sec_adv file.

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Related Topics