FWIS3

Here's one for squid (and maybe FZ)... What are some good textbooks on x86 assembly? Preferably with an eye towards disassembly - reverse engineering, malware analysis, that sort of thing?

(I know this is not too relevant to the sysadmin path, and it's more of a hobby for now. But it's something I'm starting to get into.)

IMHO a good sysadmin should be like a sponge and absorb anything that helps get the job done even if not directly related to what a "sysadmin does".

In Larry Wall's "Camel Book" the first line of the book is:

Larry Wall wrote:Perl is a language for getting your job done.

Which might explain why so many sysadmins scribble in Perl.

And no, I really don't have any good resources on x86 assembly programming.

You'll find better and more up to date resources online now than in books GJ.
Programming assembler is pretty easy when you are doing very very simple things.
Like writing a few lines of code to manipulate I/O etc
However, writing assembler to interface to the system gets more involved exponentially the deeper you go.
e.g. interfacing with C requires the use of stack based parameter passing.
While the interface is pretty straight forward it gets more convoluted because systems provide for different methods to pass variables rather than just the stack. They can pass stuff via registers as well specifically for speed/compactness.
You have to also write code that uses the stack to cater for re-entrancy which plagued the early PC with non re-entrant DOS calls and any routines that use these also became non re-entrant.
With the PC and its x86 architecture you also start getting into segmentation even though it does support linear memory models. This is because of backward compatibility issues. Then there are the 16/32/64 bit modes and you have to know which you are operating in.

Needless to say, the dis-assembler becomes a complicated bit of work where it needs to understand all the variations in the long list of CPU derivations. Its not like the good ol days where you had one instruction set for a family and it stayed. The PC's instructions set has been piggy backed for decades. In all seriousness, some instructions go back as far as the early 70's with the 4004 CPU. They have just been expanded in width.

So I guess to answer your question you really need to determine what you want to get out of learning assembler code for the x86. If its to be able to analyse hardcore virus/security issues then its time to give up. I say that in all honesty because to be really good at that you'd also need to be proficient at hardware at the microcode level and discrete digital logic. If you want to be average in the field I guess you could skip the digital part. The microcode level is pretty important to understand however as there are ways to force instructions to do illegal, though predictable instructions that can't be dis-assembled. It also helps when understanding just what the hell is happening inside the piplined que.

If you just want to understand x86 assembler and know your way around it without becoming an expert in the field then by all means go for it. I'd start by understanding BIOS function calls and DOS function calls and interrupts.
That will make a significant jump in understanding assembler and how the real mode of a PC works.
It also makes understanding how small assembler programs can be quite effective as a virus.

If you are interested in the security field you should start with studying BIOS Int 13h calls and DOS Int 21h calls.
Interesting reading and will give you insights into how assembler could be used very easily. Many of the early viruses made extensive use of these calls. Nowdays they are monitored pretty well by both the OS and the security software you would be running. It will however give you a good jump into understanding how these things worked and how they still work.

Hello Dunning-Kruger effect!

Thank you very much for the input FZ. I was learning this stuff fast - too fast, deceptively fast. It's good to be reminded what my limits are. I'll try to slow down a bit.

(Also good to be reminded that I'm behind the times. The malware I was looking at this week was a copy of TDSS, a boot sector trojan that's probably a bit archaic by now.)

I take it, re security stuff, it's possible to pull a fast one on a CPU via e.g. segmentation issues, in ways a novice just looking at syscalls might miss?

Re "expert" status, I don't consider myself an expert at anything yet, and I probably won't for a while. That said, Linux tech support stuff is much more my field. I doubt I will ever do serious malware analysis in a professional capacity.

BTW, at my workplace one of the sysadmins mentioned a hackathon that was coming up, which would be a good opportunity for me to learn more about C and assembly. I'll see if I can't participate on some level. But for now, I guess I'll go back to working on Puppet and such.

You're welcome GJ.
I hope I didn't discourage you too much as learning assembler takes understanding a computer and its systems to a new level. Even though its a level that very few actually need to operate at to be proficient at any aspect of computers except issues relating to hardware and low level interfacing.

It was pretty easy to fool the older CPU's wrt segmentation because they were introduced before malware became so prolific. IOW, it wasn't really fooling the CPU, it was just using the instruction set that was available to perform things that the bypassed the basic security systems that were in place.

To look at it another way, back in the golden days of assembler, when real programmers all knew it, the instruction set and the protections were basically fully accessible to the assembler programmer. It was standard that if you were programming assembler that you should have access to every aspect of the CPU and its systems. After all, that's why you program in assembler.

With the introduction of multithreading/multitasking things became less defined. Now instead of one program that runs in its own little universe, you have many programs running and they needed to be protected from each other such that a failure in one doesn't cause failure in the others. This lead to protected systems and the introduction to system level and user level privileges. Naturally this expanded as time went on and with the discovery of malware more importance was placed on the different layers of security in the CPU.

Oddly, even before the first virus CPU's had the hardware resources to protect themselves from them. They were simply never used by the OS because no one imagined malicious code would be introduced deliberately, I mean, what's the point? We know better now.

The segmentation process was introduced partially to keep backward compatibility and partially to provide the means to run many processes separate to each other and partially to enable extremely fast context switching between programs. It was marginally successful at all of them and it also placed limits on the size of the programs running in each segment. Initially it was contained in several memory models defined by the segment registers, 64k, 128k, 192k and 256k. Most ran more than 64k but the 64k segmented programs were extremely popular due to simplicity. Any xx.com program was fully contained in one 64k segment for example. xx.exe programs could be any size but they were relocatable in address space as well having part of their file filled with the required relocation information.

A xx.com program that modified the segment registers was thus obviously flawed or malware. This became very easy to detect for them. The xx.exe programs were more difficult as they actually were designed to be able to modify the segmentation registers and thus increase their data space on the fly beyond the four memory models I wrote about earlier.

Things are a bit different now. When a legacy program is started by the CPU it is within a self contained area and if for any reason it tries to access memory outside this area it triggers an exception routine to the OS. I won't start on the poor practices of exception handlers in the early days other than to say that the writers of the OS's were more high level programmers by that point and didn't comprehend the implications of what exception handling meant. The result was some superbly unstable and unreliable systems. Needless to say, this has since changed significantly and the writers of OS's understand more about correct exception handling than the hardware guys who designed the handlers and the assembler guys that used to use them. This is because they had to learn to deal with a world changing beyond what the hardware guys ever imagined. Thus was security born into the world of the PC.

Anyway, while direct segmentation manipulation by malware was pretty easily prevented, the malware writers didn't have to look far or long for a new attack method. OS's provided hooks into the OS that performed all manner of delightful things that any self respecting malware programmer would salivate and drool all over themselves over.
Because everything is interconnected (the big selling point) we came to get activeX, javascript malware etc.

Why deal with pesky assembler and low level instructions when you've been given high level access via alternative sources. Enter the world of the script kiddies because just about every package to survive wants to provide access for plugins and running user code and make the market happy.

Now in a complete about face, we have gone back to assembler and low level hardware attacks but this time they are being done by seriously gifted programmers with hardware backgrounds. Enter the age of malware designed to infiltrate specific sites, PLC's controlling large factories doing things your government doesn't want other countries factories doing. Attacks that use state of art encryption, attack at multiple fronts on totally different systems that require expertise from many different fields to successfully accomplish.

That's where we are now.

The next phase is the commercialization of the attacks. When you have a competitor in a multibillion dollar industry its extremely cost effective to sabotage their factories and R&D leaving you with a short term monopoly in the market. That's my prediction in any case and has been for a while. Expect to see certain companies being attacked by supposed "crusaders" of the planet but know its more likely the new trend in business. Capitalism practicing its inevitable conclusion.

Does that help any? lol