More hot emulator action

After seeing a random Facebook comment from a friend, I spent quite a few hours this weekend working on my NES emulator to get the game “Low G Man” working. Turns out the issue was related to the fact that an NMI (Non-maskable interrupt) can occur mid-cycle, and that particular game waits around for the VBLANK (which is what generates the NMI) to occur (other games do this as well). Unfortunately, because my emulator doesn’t really support intra-instruction events (the emulator is single-threaded), the loop waiting for the VBLANK would never see it.

Essentially what I think was happening was

; wait for VBLANK
LDA $2002
BNE ...
(let PPU run... VBLANK occurs... causing NMI)
(NMI interrupt handler runs... which reads $2002, clearing it)
LDA $2002; this now doesn't see the VBLANK because the NMI handler cleared it
BNE ...
LDA $2002;
BNE ...

I added in a modification that the PPU (or anyone) could tell the CPU “hey, this NMI actually occurred a litte later than you think”, allowing for one extra instruction to execute. Which, in this case, is enough to get it out of the infinite loop.

So, effectively it now does

LDA $2002
BNE ...
(notify PPU to do work, generating VBLANK + NMI)
LDA $2002; see the VBLANK
(NMI handler runs, but the LDA has already had a chance to run)
BNE ...
; yay, outside loop!

I had previously done a hack of setting the “sign” flag (which is what the BNE is actually looking for, because it is that highest bit) in the CPU NMI callback, based on a recommendation of something I found. But this felt like a horrible hack. I’m not certain that the new way isn’t a hack as well, but I think it is at least more “realistic”.

On another note, I’ve put the emulator into github:
https://github.com/jdewald/qjnes

I couldn’t think of a name (JNES is already taken), so I just did Quay’s Java NES emulator (qjnes). Note that this is still technically both a C64 as well as an NES emulator.

If you do happen to download it, you can fire it up with ant:
ant nes -Drom=

While you can play a lot of games with it (Super Mario Brothers 3 plays well, Kirby’s Dreamland played well last time I check)… there is no sound and many glitches (it is remarkably how ‘perfect’ some games require the system to be). This is not the emulator you want to get if you actually want it for “fun” (Nestopia and the likes are for that). It was purely so I could “see if I could”.

It doesn’t do anything cool like bytecode manipulation and JIT compiling… because it doesn’t need to on a modern computer. I finally added code to sleep, because on my Macbook PRO it runs well over full speed, so I now try to keep it around 60fps.

Interesting bits about the JVM

Here are some random things about the JVM that you might (or least I didn’t) not know:

• boolean is represented as a 4-byte integer internally and treated as such in all bytecode-level operations (method parameters can be specified as being booleans)
• Often when a NullPointerException is thrown, the JVM actually has access to the method that was being called, but the NPE is generally thrown in the calling method, rather than the called method.
• Reflection is really inherant to how the JVM is specified and operates; all methods are located by {class,method name,method parameters} dynamically
• The JVM truly has no concept for the Java language; String and Object are for the most part the only classes treated specially
• The JVM is stack-based rather than register-based

Some cool things about C# (.NET as a whole?):

• C# has the notion of Properties, which let clients of your class access members “directly” while at the same time allowing for change to underlying implementation (as they are actually getter/setter methods)
• C# allows for the notion of first-class methods, which it calls delegates. This lets you avoid defining a whole interface to get access to a single generic method.
• C# has first-class support for firing events, which makes use of the delegates (not sure if it has to or not, I’m still learning this stuff)
• C# requires you to use override prefix for a method if it is overriding a parent method, which makes it clear to readers of that class that this is in fact overriding something.

I learned these little tidbits while learning C# by writing a JVM in that language. I am fascinating by emulators and VMs because they are software that represents something that is well-defined (http://java.sun.com/docs/books/jvms/) . So in a sense the (naive) implementation is fairly straightforward, if tedious. (I also learned that you can actually write a non-trivial C# application by effectively writing Java and renaming it with .cs and doing a few basic replacements… or just about.)

This project would not have been at all possible without the GNU Classpath project, which has worked tirelessly to implement all of the standard Java classes as well as reference implementations the classes needed to tie to an actual Virtual Machine implementation. I have not gotten JNI to work yet, so for the time being I am implementing the native stuff directly in C# (which makes sense actually as that is my JVM implementation language and is sitting “below” the JVM).

Both Classpath and the official Sun jre implmentation (which is now open source as OpenJDK) provide real world implementations of that stuff you have not looked at since university (Hashtables, Linked Lists, etc) in a fairly readable format. And because they are real world, they offer glimpses into the optimizations and workarounds that have to be done to make these data structures work in the real world.

There is also a project called IKVM which is a very complete .NET-based implmentation of the JVM as well as the class libraries which allows for .NET applications to actually execute Java classes. I think it includes a combination of GNU Classpath and OpenJDK classes inluding managed .NET implementations of the native methods. If I continue this project (not terribly likely, I don’t call it “ToyVM” for nothing) I will probably migrate to using that so I can focus on the internals of the JVM itself. When I started I just wanted to get going and I had issues with the version of Visual Studio .NET that I had and then could not get GNU Classpath or OpenJDK to build or work in cygwin. I actually used MonoDevelop to do the C# development (so I wrote a JVM in C# using a Linux-based .NET implementation running on x86 VM on top of Windows XP).

A couple of nights ago, I finally got the “Hello, World!” application to work after about 2 months of development on and off, not sure what the actual man-hours was.

I used C#’s event handling/delegate set up to do gather some runtime statistics for the basic “Hello World!” application and apparently it loaded 142 classes before it finally did the output. Most of these were not used obviously, but are part of the environment that is statically loaded by key classes (like Charsets).

Next steps:

• Make it look more like C# (Cxx languages tend to use GetBlah() rather than getBlah(), and C# supports Properties which I would like to make use of)
• Implement Garbage Collection (pluggable perhaps). I am curious about the various methods that are used and which are better in various situations
• Do some additional refactoring into additional namespaces
• Optimize the most heavily used bytecodes if possible
• See what breaks when I run things other than HelloWorld.class 🙂 I am very much a just-in-time developer so I only got the stuff working that were absolutely required to get Hello World to work. 84 byte codes have been implemented (out of the 107 that were encountered, but some load xload,xstore,if,if_icmp get reused with different initialization parameters)
• As mentioned before, look into integrating with the IKVM libraries so I can worry less about the native aspects of it

I have put the code into a local git repository, which is another tool that I have been wanting to play around with. I am happy to push that out somewhere, with the normal caveat that this is your typical homebrew code that is not as well commented as it should be (but is hopefully well structured enough for it to make sense).

Happy Coding.

Schools and that jazz

There has been a lot of furor (at least from all the links popping up on proggit) around the “worth” of CS degrees and how bad the programs are.

My personal take is that people are expecting the wrong thing out of it. There is certainly a mechanical/trade aspect to programming. That’s the part that they can teach quite well: syntax, basic algorithms, etc. Strangely, this is the part they only teach in Software “Engineering” courses versus pure “Computer Science.” Most people go into these programs expecting to be able to walk into a typical business programming job and get to work. They really do no want to learn about Big O, Finite State Machines, or Data Structure Implementation. Who wants to know about all that damn math!?

The bit that they do not teach well is the part that actually makes you good at “programming”: critical thinking problem solving skills. Much of your time is spent figuring out how to go from problem to solution and, after doing that, why the apparent solution does not actually solve the problem correctly. You will spend a lot of time debugging software and fixing bugs. That’s just the way it is. Yes, each language makes some aspect of expression easier, but at the end of the day the actual algorithm is exactly the same. There are really only two ways a bit of program can be wrong:

1. The algorithm is incorrect
2. The expression of the algorithm is incorrect

I would argue that the analysis of either of these problems requires slightly different skills. One of them is the heart of “computer science” and it is the creation of algorithms that solve problems in faster and more innovative ways that are researched. The majority of us will never come up with a truly new algorithm; rather we will solve a problem that is just being defined in terms of different nouns. So a key skill of any programmer (during the design phase of construction, however short that may be) is recognizing how the problem can be re-phrased in another light and use a known algorithm. The site TopCoder is an excellent way to practice this.

Assuming that the proper algorithm has been chosen, the next step is to actually implement it. Theoretically this is the “easy” part, but it is also where the majority of effort is placed in the real world. An absolutely necessary skill of a software engineer is to be able to follow the logic of code (usually people speak of reading code but I really think you follow the logic of code instead. While I have seen some poetic code before, it really isn’t literary in nature) and trace what it is doing with a particular input. It is this skill (or the lack of) which is why, i believe, people complain about bad Computer Science education. You can whine and moan about Java or C++ being used (instead of “pure” languages like Haskell) but frankly that is a bunch of hogwash. If a person is getting the right education, or has the right innate talent, then they will be able to solve problems in any language given to them.

I have always said to people that one of the most useful classes I ever had in college was my Physics class. The professor was smart and did not allow calculators on the exams. You see, it is not the answer that matters, but how you get there. Your algorithm. The most important lessons in Computer Science (and medicine, and law, and….) is those that teach critical thinking and being methodical about solving a problem.

Ruby will not make you magically a better programmer. Java does not turn you into some brainless idiot. Perl will not turn you into a person incapable of writing clear code. Using RAD tools will not prevent you from learning how your code actually works. It is the person behind the code that matters.

Update: I found this response to the debate by Brian Hurt at Enfranchised Mind to be very good (and much better written than mine) in the sense of mentioning that, effectively, you want a “Developer that knows Java” rather than a “Java Developer”.  The reality, though, even if we don’t want to admit it, is that companies want Java developers. What do they care if the person will be useless 10 or 20 years from now, they’ll just get a developers that are trained up on New Fangled Language X.

Holidays and a New Year

It’s been a while…

I’m ending a nice long 3-or-so week extended Christmas (just me, the wife, and pictures of gifts from family), New Year’s (very cool dinner on a ship permanently docked on the Thames. Those who know me won’t be surprised to know that I spilled red wine all over the table within about 2 minutes of sitting down. The 9 others at the table were quite nice about it) and 5-day Moroccan holiday in Marrakesh (مراكش). The last was quite cool (finally something different from Europe, you can only handle so many castles) but hectic and wearing at times (I can only handle so much haggling.. even though it’s satisfying to only pay 50% of the original price, I know I’m still paying way more than the price a local would pay). Again those who know me will not be surprised to know that I dropped (and shattered) the large tagine that we had purchased… was about 10 feet from our boarding gate at the airport.

And to really but an end to the holiday, my wife is now on a plane back to the States to get us going for our repatriation there. I will be following 3 weeks later, as my year-long stint here in the UK is ending. I have have had an awesome time here, both at work in out and about. Met some great people who I will definitely miss.

And now for something completely different..

To bring things back around to geeky stuff (I tend to skim over other people’s personal stuff, so I understand if you, Reader, have done the same) I have finally started working on my Super Nintendo (SNES) emulator. It is still using the same basic core as the C64 and NES emulators. Main difference is that the SNES using a 65816, a successor to the 6502 which can address 16MB of memory (24 bits) and implements all of the 256 op codes and adds some more addressing modes. When it initially starts up, it is actually in 6502 emulation mode (with bugs that the original 6502 had fixed, which I’m sure provided frustration to many developers who depended on undocumented instructions and bugs). I have gotten some basic graphics to work in the ‘test.smc’ demo included in Y0shi’s documentation, but it is nowhere near even able to get screenshots, but hopefully only a week or so (I’ve spent a feverish 3 or 4 days dedicated to SNES stuff, but probably spent another couple of weeks previously working toward getting an Apple IIGS emulator working, which uses the same processor) to get there.

I have started adding some JUnit tests of the actual instruction implementations, as even minor bugs can truly spiral out of control in the running of the system.

As usual, Zophar’s domain has proved invaluable for obtaining documentation, but I have also used Wikipedia for system overview (memory sizes and the like) and another site I stumbled on just called “Emu-Docs”

I will make the code available via CVS or Subversion once it is in a usable state. Apparently my wife never really played the SNES, so we shall see if I can find anything to drive me like getting SMB3 working on the NES did.  I would love to get Super Mario Kart working.

I have been using ZSNES as my “reference” for what a working system should look like (I don’t know if it’s open source or not, but I am only using the executable in any case).

Shoutout goes to the many people who dedicated hours and hours dissecting and analyzing the SNES and writing emulators in C and Assembly which ran on old Pentiums. My Java emulator may one day run at a full speed on a 2 Gig machine 🙂