Monday, April 12, 2004

Console app in VB 6.0

Recently we had a requirement where vb apps need to be compiled as console applications, so that they can be executed by a java application. The success or failure of console vb app where to be notified by returning th exit code. Returning the exit code wasn't tough, you can make use of ExitProcess API to return the exit code. The problem with ExitProcess is that if you call it within the VB IDE it closes the entire VB. So we used the Debug.Assert hack in the module to figure out whether the app is running in IDE or not. Everything was going on well, till one fine day I decided(can't stay out of the trouble!) that we need to make the application more "OO" and hence thought of moving the console helper module(the module contained ExitProcess and support for ConsoleWrite and ConsoleRead) to a COM dll. So created a dll, included the module and added a wrapper class module, which would just delegate the call to the bas module. The first problem that I faced was figuring out whether the app is running in debug mode or compiled mode. Debug.Assert won't have helped, as the dll was always compiled and compiling the dll removes all Debugging code. As I have learnt in my life as a programmer, when in doubt turn to google, and guess what? found a link to vb2themax article which solved the exact same problem. So included that piece of code and everything went fine except whenever an app would call ReturnExitCode on my consolehelper dll, it would crash with a "Stack Overflow" exception. After coupla hours of debugging figured out that there is a bug in VB compiler(or so I would like to call it!!), if you have two methods with the same name & same signature one in a bas module and another in the class module, the compiler compiles the dll without any complaints. In my dll, I had included the bas module which had a public method "Public Sub ReturnExitCode(byval ExitCode as long)", and I had copied-pasted(the same ol' programmers habit!!) the same method in the class module. The code in my class module was like:

Public Sub ReturnExitCode(byval ExitCode as long)
If Not InIDE Then
ReturnExitCode ExitCode '''this call was supposed to be made to the method in bas module
End If
End Sub

Obviously the code goes into endless recursive loop and throws a stack overflow error. I know it was oversight on my part but still feel vb compiler shouldnt have allowed having two methods with same signature at first place. The fix was pretty easy, rename the method in bas module to something (I chose ReturnExitCodeInModule!). Problem solved, should have relaxed after that but as they say a solution is mother of every new problem...
we had been converting the apps to console application by running EditBin.exe after the app was compiled, so I thought that it would be great if we could compile apps to console application directly from vb compiler. My first thought was to use one of the available add-ins like the one on, but that wouldnt have been any challenge for a programmer like me :-) so decided to go Bruce McKinney's way(btw, mcKinnery is the author of oh so famous Advanced VB 5), hence I created my own Link.exe, all that would do is interecept the command line parameters sent to it by VB Compiler, change the /SubSystem:Windows switch to make it /SubSystem:Console and then shell the original Link.exe with this tweaked cmd line params. Guess what? works like a charm, another problem solved i.e. birth of a new problem! now all the applications are compiled as console applications, so how do you compile normal windows apps?? one way to beat it was only if the app references the consolehelper component, my custom linker should tweak the command line else it should pass it on to the vb link.exe untouched(I know it makes the implementation very specific to my needs but hey I never wanted to make my custom linker a commmercial product). But to the linker vb only passes the obj file names & not the references, tried custom C2.exe but that didnt help either, looks like vb compiler itself takes care of references et al(which makes sense), 2 days gone and am still looking for the solution so that I can then concentrate on a new problem.

No comments:

Post a Comment