OS Development Skeptically
Introduction
This document has been written to address certain questions
of the OS development that often arise when attempting to make a homebrew
operating system. The document is probably of no use to those, who are
professionals in the area. So, if youre an expert in OS dev and you know the
thing from inside out, dont waste time reading this document. This document is
aimed at newbies, wannabies, and other unprofessional coders, who want to make
their own OS and have already faced (or are yet to face) a lot of trouble down
the way. The document shall give some ideas and hints as to how to do certain
things and how not to do them.
Why Developing an OS?
Right, why developing an OS at first place? What does this
give to one succeeded? What is so fascinating in doing it?
Well, there are several reasons for making an OS. To name a
few reasons and attitudes (add your own if not listed):
- Im
a fucking cool coder, much better than those kiddies from the next block.
I can code everything. An OS is a big/small yet interesting project for
me. I can manage. Lets get started!
- Im
bored with Windows, Linux, whatever. Its now time to roll my own. I cant
wait anymore for the better OS; I need to write some code. Lets start
with some fancy code that will boot up my PC off the floppy disk
- I
wanna be famous!
- Its
interesting how things really work. Making an OS is a way to learn, its a
way to improve my programming skills and extend my experience of
making/using systems software. Finally, its fun.
- I
have a plan to make an embedded device, for which Im gonna need an OS to
run task-oriented software under it.
- Ive
got a course project in OS development, need to make some OS and submit it
before the deadline.
Those, who can put their sign next to items 4, 5 or 6 of the
list, should read the document with attention. Those, who can put their sign
next to items 1, 2 or 3 should read it with double attention, or maybe even
triple. The life has proven that the humans ignorance and stupidity is
practically unlimited. So, you never know what factor should be used as a
measure of increase of the attention or how many more times one needs to reread
something to get the idea. In short, it depends. It depends because the guys
not having college or university education in computer science have very
different knowledge, skills and experience. Those, who have that education,
have at least some basic prerequisites.
And when youre through with an OS (meaning youve done it),
youll be able to say you acquired extra knowledge, skills and experience. If
your OS isnt a lousy piece of software, it may be useful for your future work.
How Hard is it and How Long does it Take to Make an OS?
Just like I said, it depends. It really depends on the
knowledge, skills and experience, which are very diverse among the people
attempting to do an OS on their own, to do it from scratch.
It also depends on the kind of the OS in mind If theres
nothing in mind, it will probably take unfortunate (if not indefinite) to make
an OS. Actually, this is probably on of the very first things that must be
decided early on, thus this contributes to the prerequisites, to which Ill
soon turn.
OK, speaking in terms of time, OS development takes months.
Or years. Yes, that much. Take my word for that or, if you dont trust me, try
and see yourself. :)
Just how hard is it? Id say the difficulty and amount of
time needed are proportional to one another. Its inappropriate to measure the
difficulty in some everyday units, especially mechanical-like ones (e.g. number
of source code lines or number of key presses per day multiplied by number of
days spent in typing :). There are problems that can only be solved by thinking
and finding correct solutions, not by writing more code, not by fixing holes in
the algorithm that is just plain wrong to work in general or in particular.
Some OS Dev Myths
- It
is easy: its actually not. If my words arent convincing, give it a try
and see :)
- No
assembly language required: it is always required. It can be as little as
1% of the whole source code, but it needs to be there to take advantage of
the target hardware (e.g. CPU) specific facilities, of which your HLL
compiler (say C/C++) knows nothing
- OOP
must be everywhere so things work nicely: the 1st point is that
you can make flexible software w/o what is meant to be OOP (e.g. in C, w/o
C++), use pointers to achieve flexibility.
The 2nd point is that if you decide to use OOP, say you go for
C++, you will have to be very careful about writing any OOP code because
it relies on the proper runtime support of the OOP functionality (memory
management to store classes/objectects and exception handling, to start
with). You must either provide this runtime support for these OOP things
or refrain from using them altogether. Unstable or buggy OS kernel is of
little to no value You might find that it is wiser to make the kernel
core in say C and ASM and leave OOP to the user applications and things
like a GUI. The problem with runtime support for OOP (in case of C++) is
that this runtime support isnt standardized and may differ not only from
compiler to compiler but also from one version to another of the same
compiler.
The 3rd point is that if you want your OS to be OOP from inside
out, you must understand where you can have benefits from using the OOP.
Just declaring the OS as an OOP OS doesnt make it any OOP-like. Using OOP
blindly doesnt either. First, find areas where OOP is really beneficial
for the sake of the OS and user applications. Figure out all those base
classes and abstract APIs, which will make the thing flexible, modular,
structured, reusable, extendable, etc etc. Use OOP where it is beneficial
in terms of functionality and where it simplifies programming without
making any trouble.
Ive seen projects with extensive use of C++ and OOP where they were not
need, helped little in solving the task in hand and actually made the
source code less readable.
Im not against OOP. It just has to be in the right place and amount, just
like ASM.
There should be more myths, but these are just the ones off
top of my head. There have been many posts in OS dev related newsgroups about
these.
What are the Prerequisites and Desirable Background?
If you think
education is expensive, try ignorance.
- Some smart guy
- Desire
to do an OS
- Passion
and patience
- Imagination,
open mind
- Ability
to work independently and actively search for solutions, not relying on
the help from anyone else. Ability to work in team is also a good thing
- Self-control
and self-organization
- Math
skills (calculus is probably not necessary, but it may be useful)
- Problem
solving and logic reasoning skills
- High-level
language skills (such as Pascal/Ada or C/C++ and the like, but forget
about Basic and HTML/JavaScript!)
- Assembly
language skills (I really mean it)
- Knowledge
of the digital logic (including Boolean algebra, deMorgan laws,
understanding of finite state machines, see for example the book by Brown
and Vranesic)
- Knowledge
of the basic computer architecture (arithmetic and logic unit, memory
organization, caching, bus organization and operation, program/data and
I/O address spaces, I/O capabilities and interrupt servicing, etc; see for
example the book by Tanenbaum on this, etc)
- Knowledge
of the fundamental algorithms such as searching and sorting (not limited
to these only) and corresponding data structures organization (see e.g.
Sedgewick, Knuth, etc)
- Knowledge
of the basics of the OS theory (e.g. memory management, multithreading and
scheduling, priorities, race conditions, synchronization primitives,
deadlocks, file systems, networks, device drivers, terminal/console I/O
and GUI, etc etc; see for example books/docs by Tanenbaum, Finkel, etc)
- A
basic OS plan, e.g. overall structure, required functionality and
performance plus some ideas on how to do such and such part
If anything of the above is missing, you will have two
options in general:
- Quitting
- Learning
or catching up and keeping going
Many guys had done some lousy work and then quit because
they were not able to continue due to the time limitation or just ignorance,
which they couldnt overcome.
Just dont overestimate yourself. Face the reality. You will
need to learn a lot of stuff to make your OS dream come true. I started
programming at the age of 12 in Basic. When I was 13, I learned the Z80 CPU
assembly language. I got an x86 PC when I was around 15. I had been programming
in Pascal and learning x86 CPU and x86 PC hardware until I turned 19. Then with
newer (faster) PC and newer interests, I started learning C and doing some OS
dev related stuff. Im now 26 and Ive been doing all that OS dev stuff for 7
years now, whenever studies and work allowed to. Meanwhile, Ive begun paying
attention to the fundamental algorithms and performance because its important
for the code to be not just correct but also efficient to a certain degree.
What else? I havent mentioned my background. A few words about it At school I
was attending a math intensive class. I studied physics at university, so its
again quite some math, not to mention the importance of physics and
engineering. Later graduate studies at an ECE dept helped me with computer
science even more. Its a good background. Math, physics and engineering wont
do any harm, quite the contrary. But all in all, Ive been programming for more
than half my life now (14+ years) and I havent made an OS. I must be having a
lot of knowledge and experience but I still dont have an OS. Why? :) Because
its not easy to make one, because its time consuming and because I dont
wanna end up with some crap that can hardly be referred to as an OS. That
simple. And so far, it all mostly has been just for fun and education :)
However, recently, it has begun to help me at work. If anyone is still
questioning my occupation, Im a full-time software engineer.
What an OS is and what its not
Well, an OS is a piece of systems software, whose primary
purpose is providing user applications with easy ways of communicating with
various computer devices and each other. The ease is achieved through hiding
all the hardware specifics from the user applications inside the OS (normally,
inside the hardware device drivers) and providing a unified API to the
applications for accessing the devices. Nowadays, we have millions of different
kinds of computer devices. It is impractical (if not impossible) to incorporate
support for each and every device into the user applications, which should run
on millions different computers with different hardware and performance. And
this task has naturally moved from application developers to the OS developers
and device developers and made user application development easy (well, easier
than in the times, when the computers had no operating systems and thus
software engineers had to write hardware specific code for every computer on
which the application was meant to run). These days an application can run
without any changes on practically any computer. This is a good thing.
Now, when someone is saying that their OS boots up, enters
system/kernel/supervisor/privileged mode (the protected mode on i80386+) and
handles display and keyboard I/O and then this person asks for help with
something, all I wanna do is laugh. It is just ridiculous for some hello
world program to be referred to as an OS. It has none of the functionality
that an OS must naturally provide to user applications. Null, void, zilch. Yes,
it can be very fascinating for a newbie to watch their program boots up and
prints something to the screen. But wheres an OS? That program is nowhere near
to an OS and it wont become much closer to such in the near future, say, in a
few weeks or months. One could argue that there may be small OSes with little
of functionality and blah, blah, blah. Fair enough. Make something like MSDOS,
which is, basically, just a file system, a memory manager, a few I/O drivers
and a collection of programs/tools/utilities, most of which are used rarely or
never used at all. OK, make it like MSDOS, if its what you want. Make it have
a file system and memory manager. Thatll be enough for a tiny OS. Yet, many of
the coders concentrate on anything but the things that will enable them to have
at least these key features. They hurry up making bootable sectors,
bootloaders, etc etc. All right, Ive done this all boot stuff as well. But I
wouldnt give an OS name to a bootsector. It is just not it.
Where to Start and Where not to
Many coders rush into coding something. Like it has already
been mentioned, many start with the bootstrap process and protected mode. Its
of no surprise to hear that they dont know what to do next. And many of those,
who havent done anything yet, dont know where to start at all.
All right, starting with some hardware specific thing is
good only if this specific thing is gonna be used. Its also good if one has
never done anything similar before. No other good reasons exist for starting
with hardware specific stuff. It is just a detail, which will be different on a
different computer, especially on a totally different hardware platform. This
detail should be hidden inside the driver and not shown to the entire outer
world.
Lets put hardware and drivers aside for the moment. Then
what remains? Surprisingly, lots of important things. If we consider any OS
function, then well see that theres some idea behind it, some algorithm.
Well see that there should be certain structure and behavior of the system and
its smaller parts.
So, where to start:
- In
the OS one will need to manage various kinds of data in different ways.
Here come lists, trees, FIFOs, stacks, caches, hash tables, etc etc. Find
out where youll need them. Therere many places, so why not start with
these?
- Every
part of the OS needs memory, for code, for stack, for static data and
heap. Why not write a general-purpose memory manager, which youll extend
later if need be? The memory manager can be entirely developed under any
OS (DOS, Windows, Linux, whatever). Just allocate some memory by standard
means (say, Cs malloc()) and manage this allocated memory by your own
memory manager.
- If
you want a file system in the OS, you can design and implement one. And
you dont need to perform low level (sector-based) disk I/O for that.
Emulate the disk with your file system inside a file, under a working OS
(e.g. DOS, Windows, Linux, whatever). Dont try to make it work under your
incomplete, unusable, and buggy OS.
- If
you want a console/terminal, window manager and GUI, Im pretty much sure,
its also possible to develop things using the existing display I/O
functionality of the host OS (e.g. VGA and banked VESA modes can be used
under DOS/Windows, and text I/O can be done under any OS). And dont
forget that you can always create a file, to which you can save a picture
instead of drawing it on the screen. If you store it in some standard
graphics file format (.PCX, .BMP, .GIF, etc), it will be easy to view it.
Its probably not all, but by now you should be able to see
the picture, Im trying to render.
Hints and Recommendations that May Prove Useful
- Develop,
test, debug and bugfix as much code outside the OS being written as
possible. Make this code working under the OS in which you normally work
on your PC (DOS, Windows, Linux, whatever). It will make your life much
more easier.
- For
every module you do, make at least a small test program. Run tests to
ensure correct work of your code.
- Split
big pieces of code and big tasks/problems into smaller ones. I mean it!
Attack them separately from each other. First, make these smaller parts
work. Then put everything together and correct any remaining
interoperability problems. If you do not do so, youll fail because of
being unable to locate a bug, isolate the erroneous portion of the code
from the rest.
- If
the code being developed is still big and complicated by numerous
condition checks (if-then-elses, switches, etc), make use of the state machine
concept. It often helps. It will help you to do practically any kind of
parsing and implement various data/control protocols.
- In
the code, check error conditions, dont let the errors accumulate and
propagate further than allowed, catch them and report of them by say
logging. If you dont do it, your system will most likely have hidden
holes bigger and severer than in famous commercial OSes.
- Dont
underestimate the power of such tools as a pen(cil) and paper. Dont rush
into coding something complicated. First, analyze it on paper. Draw
pictures, diagrams, solve equations (youll need surely to solve equations
in OS dev), etc. Make the solution to your complicated problem clear with
all the illustrative means you can have. Keep these papers along with the
other project documentation. If you have an idea, you can always implement
it in code and have a product. If you only have some unfamiliar/forgotten,
unreadable and probably incorrect code, you wont make a product out of
it. Keep the solutions. You can even photograph your papers and put it
under source control! :)
- Structure
and comment your code to make it possible to maintain in future by you and
others. Even the primary developer can forget what and why (s)he did in
the code. Dont make it happen. Document the code by outlining the
algorithm behind it and keeping this document along with the code.
- Read
documentation (on hardware, on algorithms, on everything) carefully.
Reread it, if you dont get it. Really do it. Dont give up. Try finding
docs that explain the same things but easier. Discuss things on the
Internet. But dont expect anybody doing your work (especially homework
assignment :) as nobody is obligated to help you with everything and in
unlimited amount.
- Use
some version control software (like CVS, MS Visual Source Safe, etc) to
keep track of the previous versions and development history. This will
help you to roll back or find a new bug that wasnt there in some of the
previous versions of the code.
- Use
PC emulators (such as Bochs, VMWare, Virtual PC, etc) to avoid unnecessary
hard disk data damages and simplify testing (if you run your code on a
real PC, the PC is hardly useful when its busy running your code, whereas
with emulated PC you can do some additional work in parallel to booting
and running your OS).
- Make
use of the map files generated by the linker. It will help you to see
incorrect code/data placement, wrong addresses, right addresses, function
addresses, etc. It will also help you to find a place of exception inside
your OS by the address of offending CPU instruction.
- Make
use of disassemblers to check that the addresses used in the instructions
(in the OS image file) are OK. Many newbies have problems with addresses
when setting up protected mode. Im happily using HIEW and BIEW as file
viewers with built-in hex editor and x86 disassembler.
- Dont
burn the bridges! E.g. dont put the code that switches the x86 CPU to the
protected mode inside a small boot sector. Dont or you will loose all
disk I/O and wont be able to load the OS or kernel (disk I/O isnt
possible in the x86 protected mode, this mode is not supported by the BIOS
that offers disk and other I/O functions). The same applies to the file
system. If you dont want to waste too much time, dont reinvent the
wheel. Make a driver for the existing commercial file system (like MS
FAT12/16/32) and avoid all problems of transferring data to and from your
OS.
Theres rationale for each and every item on the above list.
Try to understand it. It all will only help you to get done.
In Conclusion
Just dont get me wrong. Im not against anyone trying to
make their own OS. Nope. Im simply trying to point out a few key moments that
one should think of when doing it. If all what Ive been talking here sounds
gibberish, then it simply means that youve not understood a bit of it. This is
not my fault, maybe not even entirely yours, but it simply means youre not
ready or you misread something. Reread the thing. Give it some thought. Let
things settle down a bit in the head and then return to it. If it still doesnt
help, as usual, you have two options: quitting and learning. First is simple,
though you may not have a will to quit and be a loser. Second is tough, but the
more you learn, the closer you are to the success and the more chances that
this my article will be understood in the future :)
December the 13th,
2004
Alexei A. Frounze