Jungle Coder

Color themes are out!

Color themes have been added to junglecoder.com! For now, this means that different articles can have different color themes set, to add a little flavor to things.

published Sat Nov 11

EVA_voice: "Jungle Coder 2.0 Online"

Who?

yumaikas, aka Andrew Owen, aka me, your proprietor of all things junglecoder.com

What?

This is the second major version of junglecoder.com as a website. The old version was in Go, and while it had a venerable inspiration for the layout/design, I’ve progressed as a creative, and have new ideas and goals, and inspirations (don’t blame Xe for my color choices, tho, they are my own).

When?

4 coding sessions over the last two months from 2023-7-20. A multitude of various pokings at Phoenix/Elixir.

Where?

On the internet, for edification and entertainment!

Why?

Because I want to have fun! Because I don’t want to write Go to maintain my old site. Because I want to do stranger, more cursed, less corporalegible things with this particular site. Like give every post it’s own color scheme, or build a set of arcade pages for the various games I’ve made, or

How?

A confluence of thinking about it, and then doing it off and on. There’s more I want to do, but this layout overhaul is a start.

More concretely, wsl2, iex, Phoenix, SQLite and some late nights

As the Advent is Drawn, 2021

My wife and I wanted to do a low pressure advent challenge, since creating this Snow Globe was so fun. She pulled together the challenge below. For anyone interested in following along, I’m planning on posting on #astheadventisdrawn on Twitter and Instagram.

Week 1

Day 0, 11/30, preparation

The first prompt: Gingerbread Men/Houses. I used Janet to roll for the mix-in prompt, and got C, aka Candy Canes

repl:7:> ([:A :B :C :D :E :F] (math/floor (* 6 (math/random))))
:C
repl:8:>

Because I’m building this with p5.js, I went ahead and copied code from my previous p5 sketch. The main thing of note is the pointInTri and tri functions I had from before.

function setup() {
    createCanvas(windowWidth, windowHeight);
}


// Based on https://stackoverflow.com/a/61804756/823592
function pointInTri(r1, r2, topx, topy, width, height) {
    let [tx, ty, h, w] = [topx, topy, height, width];
    let [Ax, Ay] = [tx, ty];
    let [Bx, By] = [tx + w / 2, ty + h];
    let [Cx, Cy] = [tx - w / 2, ty + h];
    let sqrtR1 = sqrt(r1);
    let x = (1 - sqrtR1) * Ax + (sqrtR1 * (1 - r2)) * Bx + (sqrtR1 * r2) * Cx;
    let y = (1 - sqrtR1) * Ay + (sqrtR1 * (1 - r2)) * By + (sqrtR1 * r2) * Cy;
    return [x, y];
}


function tri(topx, topy, width, height) {
    push()
    let [tx, ty, h, w] = [topx, topy, height, width];
    triangle(
        tx - w / 2, ty + h,
        tx, ty,
        tx + w / 2, ty + h
    )
    pop()
    return [ty + h , ty] ;
}


function randPtsInTri(orns, params) {
    let pts = [];
    for (let r of orns) {
        let pt = pointInTri(r[0], r[1], ...params);
        pts.push(pt);
    }
    return pts;
}


function draw() {
    background(6, 6, 6);
    let w = Math.min(width, height);
    let ww = width;
    let wh = height;
    stroke('black');
    fill(200, 200, 200, 50); //234
}


function windowResized() {
    resizeCanvas(windowWidth, windowHeight);
}

This index.html document isn’t set to change during the challenge, but is shown here for anyone who might be curious.

<!DOCTYPE html>
<html>
    <head>
        <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
        <meta charset="utf-8"/>
        <script src="p5.min.js"></script>
        <script src="sketch.js"></script>
        <!--<script src="autoreload.js"></script>-->
    </head>


    <body style="background: black;">
        <main>
        </main>
    </body>


</html>

That’s it for tonight, I’ll be back tomorrow with a first attempt at coding some gingerbread kin!

My Janet Story

Since Advent of Code in 2020, I’ve been doing 95% of my hobby programming in a language called Janet, which not even I woud have predicted before that. I’d just broken my “heavily multithreaded or bust” phase by using Ruby for a few things, after having been big on Nim, which I gave up because of not feeling at home in the web-development frameworks it has. Both are fine languages in their own right, but where not where my brain wanted to be. Then, I thought I’d give Janet a try during AoC 2020. I’d tried lisps in the past. (And yes, despite what the Common Lisp community has to say, Janet is a lisp, even if it’s not a Lisp), but none of them ever quite stuck for me.

As I was going through the AoC challenges, parsing came up quite a lot, but Janet didn’t have handy Regex Bindings. Instead, it had Parsing Execution Grammars (PEGs). Okay, I thought, I’ll give these a shot. And I ended up actually being able to understand and use them, even if it was a little fiddly at times. A little personal history is in order. Between the years of 2011 and 2016ish, I’d done a lot of self-directed research investingating how to build a programming language, and one of the first steps that came to mind was how to parse things. I ended up reading about a lot of different parsing mechanisms, and PEGs were among them, but the examples I’d seen where hard to follow or understand. But, for some reason, Janet’s PEGs clicked for me. Some combination of maturing as a developer, Janet’s better presentation of the PEG concenpts, or how Janet’s PEG API overlaps with parser-combinators (something else I tried, and did semi-ok at) helped things gel in my head, and I fell in love with Janet.

Which is ironic, because since that AoC challenge, I’ve used PEGs all of maybe 4 times in 50+ projects I’ve started with Janet since. But, I’ve since discovered a lot of other things I like about Janet. The prevailing theme among all of these is that Janet is a collection slowly gathered clean choices. It’s not Innovative like Rust, Haskell, Pony, Scala, or Erlang. It’s a late-2010s lens applied to a mélange of Lua, Ruby/Python, Tcl and Clojure.

Appeal

So, what do I like about Janet? I like the syntatic simplicity. I like the fact that you can “call” associative data structures ([:a :b :c :d] 0) # -> :a, rather than wrapping that access in a get call. I like the soup of macros and dynamic variables that make DSL construction possible, which makes encoding intent much more straightforward than in something like C# or Go. I like that jpm and project.janet have first class support for building statically-linked executables and archive-libraries that mix Janet and Native modules into a single artifact, which makes building small CLI tools for Windows a breeze. I like that the Janet REPL has easy access to documentation attached to any given function/variable/module. I like that Janet makes writing C modules about as painless as that can be (some pain will remain, depending on the C libraries you’re trying to interface with). I like fibers as a collective tool for managing both asynchrony, and code “signals” (think having the ability to throw exceptions, but also use that for general messaging, as needed). I also like that Janet embraces a very clean flavor of multithreading that seems inspired by Tcl/Erlang, where memory isn’t shared, but one can send messages over channels.

All of this adds up into a language that I find very easy to write (I’ve written over 8kloc of Janet in my spare time in the past 9 months) when the libraries exist for a given task I’m doing. That does, however, bring me to some very important points about Janet.

Travails

I’ve been singing the praises of Janet for over 600 words, now it’s time to cover the rough edges. And there have been quite a few, though less due to the last year. The biggest rough edge of Janet is that the standard library is still getting bugs shaken out of it, especially on Windows, which happens to be my main development platform. In the last 9 months, I’ve found a 5 bugs in the standard library, ranging from a bug in multi-char string split, to subprocess code not actually killing child processes, to async file writing on Windows not actually writing anything other than the last line to a given file. The good thing here is that once a proper reproduction is in place, usually the fix hasn’t been terribly hard. @bakpakin, the creator/maintainer of Janet is responsive, and the bugs I’ve found, either he’s fixed in the next 48 hours, or has merged a PR containing my fix, if I’ve been able to fix them.

The other issue that has taken more of my time over the past 9 months is the fact that Janet’s ecosystem still has a lot of gaps compared to even other semi-niche langauges like Elixir or Lua. This is why I’d describe Janet as a language for hobbyists and enthusiasts, folks who don’t mind popping the hood on how things work. Because, with Janet, you’re given a language that is definitely amenable to popping the hood, but you also end up having to build more than a few of the things that would be considered “under the hood” in langauges like Ruby or PHP.

What I’ve done with it

Speaking of things built in Janet, I figured I’d end with a list of things I’ve built or helped with in Janet over the last 9 months, as a sort of overview of the sort of work being done with it. In no particular order: A polling file-change watcher called eye, (used like eye file1.txt file2.txt --cmd [command on file change]), which has changed how I approach hobby dev in a lot of ways. I’ve also helped add source mapping to C functions in the Janet stdlib (though I mostly just got the ball rolling there, @bakpakin set up the relevant macros, and @sogaiu did most of the grunt work). I written a dozen small CLI tools, including a program for measuring the size of stdin, to programs for annotating the current time onto stdin, to a task tracking program, to a program for rendering local SVN diffs in a web browser, to a program for getting/setting the dimensions of the current screen on Windows. I’ve also written a few libraries for Janet, praxis for dealing with data schemata, stringx to make string manipulation less painful, err, of which I mostly use (err/str "Message with " data " here"), but which I use often. I also helped powered by Janet get started, and was able to help it get a better project.janet parser, to better extract metadata. And more.

All that to say, that Janet has a lot of oppurtunities for buiding cool things, and learning how to grow a programming language ecosystem, but that the ecosystem is far from fully formed yet. So, if you want to join a group of enthusiastic hobbyists, Janet has a great community. If you just want to build the next feature, you’d be best to look elsewhere.

PowerShell -nologo for interactive OSX shells.

The PowerShell core team has done a great job with improving PowerShell as an interactive experience, to the point that I’ve decided to use it as my interactive shell on OSX.

This will be a short article on setting up pwsh on macOS. It assumes that you have basic shell familiarity, and have homebrew installed.

chsh -s $SHELL is how one changes which shell is used by default for interactive shells. After running brew install pwsh, you should have /usr/local/bin/pwsh at the end of your /etc/shells file.

This works well for the most part, but every time you spawn a pwsh session, you’ll see some startup text. Suppressing this text is quite possible, but isn’t directly documented anywhere.

The first part of the puzzle is that any executable file can work as an entry in /etc/shells, which means we can create a sh (or zsh, in my case) script that starts pwsh with some flags.

Then, looking over the output of pwsh --help, two flags stand out:

-NoLogo | -nol

    Hides the copyright banner at startup of interactive sessions.

-Login | -l

    On Linux and macOS, starts PowerShell as a login shell, using /bin/sh to
    execute login profiles such as /etc/profile and ~/.profile. On Windows,
    this switch does nothing.
    [!IMPORTANT] This parameter must come first to start PowerShell as a login shell. The parameter is ignored if passed in any other position.

EDIT: In an earlier version of this article, I used zsh as the shebang line in these scripts, but ran into some bug around resizing the terminal. Switching to pwsh as the wrapping shell seems to fix that bug.

Putting this all together, I ran sudo nvim /usr/local/bin/pwsh.nologo and filled the file with the following contents:

#! /bin/local/bin/pwsh

/usr/local/bin/pwsh -Login -nol

After saving that, be sure to run sudo chmod +x /usr/local/bin/pwsh.nologo so that the script is executable.

Then, run chsh -s usr/local/bin/pwsh.nologo, and add

$env:SHELL = "/usr/local/bin/pwsh.nologo";

to your PowerShell profile (edit $PROFILE from a pwsh session, creating folders as needed).

Learning without Burnout

This article is a response to the excellent question by @Nellies.

This lifelong learning thing, is there any talk on how to deal with the infinite amount of stuff there is to learn?

I do ok, but sometimes the todo list does get cumbersome.

Is that part of the whole burnout in tech? No worries, I have broad shoulders, just curious.

Initial caveats

Most of my examples are going to biased towards full stack or frontend web development, since that’s where I have the most familiarity. This is also written to people trying to break into software development, or just want to understand what it’s like to manage this aspect of a software development career. I also gloss over a lot of things, especially cultural factors that make software development harder for those who can’t pass as white males.

Types of knowledge in software development

As mentioned by Nellie, one of the aspects of software development is that software developers are expected to constantly be learning new things. Or, at least, that’s the impression that happens from the outside. There are some mitigating factors, however.

The first is that once you learn the fundamentals for a particular technology stack, moving between similar flavors of that stack is made much easier. Once you understand the fundamentals of HTTP, HTML, CSS and JS, picking up Rails, Django or ASP.NET becomes much easier.

The second is varying types of knowledge have different shelf-lives. The cool new shiny JS framework being promoted by its creator? That could be relevant for only 6 months, or it could be the next React. It can be hard to tell up front, especially when you’re new. Thankfully, most jobs also lag the hotness by a certain amount, not everything asks for the hottest new tech.

But, if you’re learning HTTP status codes, or headers and auth works? That knowledge stays relevant a lot longer, and is part of all web apps. As a bonus, learning more fundamental knowledge gives you an edge when it comes to debugging problems, which is one thing that can help you stand out in your early years. Also, different types of knowledge have different acquisition costs. Basic JS or HTML is relatively easy compared to writing a full stack web app, which is easier than building a reasonably full scripting language, which, in turn, is easier than building a production grade data store like Postgresql or MongoDB.

Another factor is that it’s possible to specialize inside a given stack. So, for instance, with Rails, if you hit an area where your co-workers don’t seem to know a lot, that’s a good place to start a deep dive into how it works. Because then you’ll be able to build a reputation as the person who can figure out tricky things. You don’t have to know the whole stack at that same level of detail. If you’re known as “The ActiveRecord Bender” or “The Database Index Guru” or “The CSS Animations Witch”, people will give you more credit when you ask for help in their area of expertise. This also synergizes with learning fundamentals, and is where fundamentals can help you out.

Tradeoffs of what to learn

There are some trade-offs in play here. The first is that most foundational knowledge doesn’t show up as keywords on a resume, and most companies are poor at assessing it (to my knowledge). Either they assume that you have it if you have relevant keywords, or that you can pick up (or have) foundational as you go, or they pattern match you during the interview against what they expect people to know for the job. And nobody has time to learn everything all at once, no, not even that super impressive open source developer or Indie game development.

Learning on the Job

So, don’t learn everything all at once. You will have to look a lot of things up during your day to day work, even after a decade in the field. But, especially in your early years, do make time to learn things. Jobs for junior candidates will expect you spend a lot of time learning things. If they don’t they shouldn’t have hired you for that level. Any programming job involves learning the companies code base for at least the first 2-6 months you’re on a project, unless they follow a very conventional structure. So take advantage of the junior position and learn your job’s tech stack as effectively as you can.

Be sure to take deep dives into the tech involved at your first position, trying to understand one layer deeper than the one you’re working at. This doesn’t have to happen every ticket, but it should be happening at least once every paycheck or two. Often the lower layers are simpler than you might think, or make understanding the higher levels a lot easier. Understanding the lower levels also gives you a much better nose for when code is fluff vs substance. This guides your debugging, grows your fundamentals, and speeds up future work.

Initial learning time

When you are in your early years, you will need to front-load learning the basics. Unless you have some other skill set that can get you into the tech industry (there are a few, Quality Assurance/Testing comes to mind, as does being an engineer in a more continuous field, like aerospace, materials design, or the like), this is going to have to come out of your own time. Bootcamps are good for getting resume keywords under your belt, though they typically don’t spend much time on fundamentals. Colleges focus a lot more on fundamentals, but might use tech 5-20 year out of date. Either one will require you to figure out how to fill the gaps left by it.

For me, I had the luck to be able to carve out school time in high school (I was home schooled) on Fridays for morning each week to work through Head First C#. At the same time, I had an aspirational project to build a Scientific Calculator GUI app, which was solidly outside what I could pull off with what I started out knowing. For the first year or two, especially before you get a job, I’d recommend setting aside a time box in a similar fashion. If life doesn’t afford that timebox, then you’ll have to be a lot more proactive about making time.

Also, don’t try to code in a void, sit down with a game plan of what you’re going to try to do or learn. If you get to your time, and you don’t have that, take a walk and think about it, or do a chore with the plan in the back of your head. Often problems are solved away from the screen, or in conversations with [ducks][rubberducking] or teddy bears. Tunnel vision is a dangerous trap in software development

Maintaining a learning pace over time

Once you have your first job (or two), and have established your base knowledge, then the question comes up of how to maintain the learning required to keep up with tech. And this ultimately depends on what you plan do to with your career. If software is a stepping stone that takes you on to the next thing, then you can afford to burn a lot more of your motivation and time, and focus where the market is hot, and try to get 3-5 jobs out of that, setting yourself up as much for the next step as you can.

If you’re wanting to stay in software for the long haul, then I’d recommend a different approach from chasing the hype. Try to find the fun in software development. And, as you learn how to find the fun in building and learning things, be sure to pace yourself. I’ve had 6-week stretches of working hard at learning a new language, ecosystem, or side-project, followed by months of focusing on other things.

You don’t have to learn at 100% every week once you feel comfortable in your ability to build projects of a small-medium size. For web development, this would probably be about the point you’re able to write your own blogging software, for whatever functionality you’re looking for. You should be learning new things on a semi-regular basis. But keep it sustainable. One thing that really helps here is to have a process for documenting what you’ve learned, in some fashion. I’ve biased towards publishing code in the past, because that’s what’s been easiest for me.

Don’t use side projects as your only tool for learning software. I used to think this was the best way to learn, because it was how I had learned a lot of things. It has disadvantages, however. For one, side projects that are actually useful are more time-consuming than something you can throw away, and the kernel of what you’re learning might be discoverable in the shorter exercise. Noah Gibbs has a lot more to say here, and I owe a lot to his writing and companionship for changing how I approach learning in the last year.

But, whether through side projects, blogging, or building a portfolio, I do think that finding a way to publish your learnings is valuable. The nice thing about publishing and shipping smaller things on the side is that you do end up building a signal to employers that you have “passion”. This also allows you to ratchet your perceived learnings more sustainably than always trying to be at 100% learning mode.

High Output times

The main time I think that it’s worth going a highly regular output (like, once a week or more often) on the side is if you are aiming to become a proper software development celebrity. This is not for everyone, and likely doesn’t pay as well as you’d think, unless you spend a lot of time figuring out how to get it to do so, and you’re magically successful with it. There’s also a whole host of Not Writing Code that goes into making money off of it.

Doing this for a season, akin to an art challenge, is a great way to learn a lot in a short time, if you have the time and energy. If you’re looking to make a lot of money on the side, however, freelancing is likely a better bet, once you have the skillset to pull that off. As much as selling your hours for money isn’t the path to higher wealth bands, it’s a much less risky tradeoff compared to product development or trying to build a personal brand.

Final thoughts

Try to find the fun, as well as a sustainable long-term motivation. Look for trade-offs rather than dogma. Use binary search debugging. Listen to your body, and be honest with yourself.

[rubberducking]:https://rubberduckdebugging.com/

Art Challenge: The Middle Grind

The story so far

Emily came across an art challenge on Pintrest, and suggested that we could both do each prompt for it.

Her medium of preference is pencil and ink, and mine is pixel art. This, unlike the previous post, covers 16 entries, because I fell behind in blog posts.

It’s also longer, and definitely has represented both Emily and I getting ready for the art challenge to be done with

Day 9: Urban Legend

Emily

Andrew

Day 10: Insect

Emily

Andrew

Day 11: Something you ate today

Emily

Andrew

Day 12: Your Spirit Animal

Emily

Andrew

Day 13: Song LyricsYour Happy Place

Emily

Andrew

Day 14: Historical Figure

Emily

Andrew

Day 15: Guilty Pleasure

Emily

Andrew

Day 16: Zodiac Sign

Emily

Andrew

Day 17: Favorite TV Show

Emily

Andrew

Day 18: Something with Wings

Emily

Andrew

Day 19: Famous Landmark

Emily

Andrew

Day 20: Beverage

Emily

Andrew

Day 21: Teeth

Emily

Andrew

Day 22: Earth Day

Emily

Andrew

Day 23: Dessert

Emily

Andrew

Day 24: Movie Prop

Emily

Andrew

Art Challenge: First 8 days

Premise

Emily came across an art challenge on Pintrest, and suggested that we could both do each prompt for it.

Her medium of preference is pencil and ink, and mine is pixel art.

Day 1: Bones

Emily

Andrew

Day 2: Exotic Pet

Emily

Andrew

Day 3: Something with two heads

Emily

Andrew

Day 4: Flowers

Emily

Andrew

Day 5: Childhood Toy

Emily

Andrew

Day 6: Eyeball

Emily

Andrew

Day 7: Crystals

Emily

Andrew

Day 8: Something from the sea

Emily

Andrew

Postscript so far

Funnily enough, Emily’s been able to to finish her pictures much more quickly than I have. I suppose I let the pixels give me an excuse to be fussy. It’s been a good way to practice working with Asperite, and to be creative and let out some of the visuals I’ve had in my head for years, but have never expressed.

A Small Command-line Productivity Tip

I really like using the command line for automating tasks. There are some things that a GUI is handier for, but having scripts handy is a very nice place to be.

One trick I’ve started using is writing small wrapper scripts (using batch files at work) for tools like RipGrep or NUnit3-console. This makes it far easier to edit those commands, and save back examples of doing different things, especially for commands with more involved parameter lists. It makes it possible to add small things, like output delimiters between search results, or to do log actions to a file or the like. It also makes it easy to reduce the total amount of typing I have to do in a given command prompt.

An example, using RipGrep, in a file called search.bat on my PATH.

REM banner is a program that spits out "-", as wide as the terminal is
REM it takes a color as its only argument.


banner blue
REM %dp~1 here is a batch-ism for getting the _first_ paramter to a batch script
rg -iw -pcre --glob "!*.xml" %dp~1


REM Other useful argument combos
REM -tcs -tsql -Txml

The nice thing about this approach is that lets me pull up an editor for when I want edit a given command and it makes using multiple arguments to --glob much easier. Were I to start using find on Unix a lot, it’d probably get a similar treatment. It’s definitely a nice thing to have on hand for more involved command-line tools.

Published Jan 12, 2020

Warming up to Unit Testing

One of the things that has been a consistent part of my software career is that I often have to approach a given idea or practice at least 3 times before I start to get comfortable with it. It’s happened with managing websites, making video games (my first Ludum Dare was well beyond the 3rd time I’d tried to make a video game), programming languages (both Nim and Factor were languages I approached once, and then again with more experience under my belt), and software develoment techniques. I got a lot more comfortable with Git after I’d had a chance to use Fossil for a while.

All this to say, that I rarely pick up something completely the first time I go at it, so for any younger colleagues that might have been impressed with my mastery of regex or shell scripting, that mastery was not achieved overnight.

Recently, I’ve started to have another go-round at Unit Testing, and this is feeling like the time-around that will have it stick into my habits as well as version control does. And, in the middle of all of this, the reasons why it seems to be sticking around seem to be a convergence of factors, not just a single thing.

For one, some sort of approach to automated testing is being required at work. Were this the only factor, I’d probably pick up a little bit of unit testing practice for work, but it would certainly not be a major factor outside of work. The other thing is that I picked up several books at the end of the year that all ended up talking about unit testing and picking the right abstraction.

The first, and possibly most influential was Mastering Software Technique, and the associated articles, by Noah Gibbs. It doesn’t have anything to do with unit testing specifically, but Gibbs consistently recommended 99 Bottles of OOP (Sandi Metz and Katrina Owen), which does have a decent amount in it about unit testing. I also picked up the book Working Effectively with Legacy Code, by Michael Feathers, mostly because I was looking for books that addressed relatively timeless topics, rather than mostly books that address version X of a given framework.

So, I ended up with a lot of programming books about unit testing. I also found in going through 99 Bottles of OOP, that the unit testing harness available for Ruby is relatively responsive, especially compared to the NUnit Visual Studio Test Explorer. Eventually, some bugs, either in my code, or the Test Explorer, led me to trying the nunit command-line runner. The difference was impressive. It makes me think that the nunit command-line tools get more attention than the test runner, because getting the command-line tools working consistently was a lot easier than keeping the test-runner working.

The last thing that seems to be cementing automated testing in my mind as a worthwhile investment was an article I ran across about Moon Pig. Often, when people tell you to adopt a practice, they don’t do a great job communicating the concrete reasons they adopted it. For me, the story of what and how various things were mocked for testability reasons in Moon Pig, mocking out the storage and time aspects of things, felt like a great starting point for how to mock things for tests, and it included a good, relatively concrete reasons for mocking things out the way they did.

So, now that I have a test-runner that works at a reasonable speed and reliability, and I’m taking in the wisdom of Michael Feathers about Legacy Code, and that I have a good story to refer too when I’m reasoning about how to mock things out, I think unit testing as a practice will be more sticky this time than it was before.

PostScript:

I should also give a shout-out to Kartik Agaram in this. Mu, and how low level he’s taking a test-driven approach to software has definitely informed how I write system-type software, as opposed to business software. PISC’s unit tests weren’t a direct result of hearing about Mu, but Mu’s artificial file system and such definitely got me thinking in that direction.

Published Jan 12, 2020

Tabula Scripta: A spreadsheet for small data-driven apps

One of my hobbies of the last two years has been writing webapps. Last year, in December and part of January, I wrote an app called Gills, a journaling tool written in Go. One feature I added to it was the ability to define custom pages for Gills written in Lua. Since then, I’ve used that ability to shape Gills in a lot of different ways.

I’ve written pages that link to the cadre of webcomics I like, pages to keep a checklist of what I need to use on caters when I was working on Qdoba, pages that gave me overviews of blog post drafts, pages that keep track of notes on a backlog, and so on.

Point is, over half of the appeal to Gills for me was having a CMS that I could script from my phone or a chromebook. The main problem with Gills-as-CMS, rather than Gills-as-Journal or Task management-lite, is that, at it’s core, Gills has two fundamental types of data: Notes and Scripts. Adding any kind of other idea means adding in a parsing layer over that.

The inciting itch for Gills was the revelation that journaling could help my mind focus. I’ve since found a lot of other things that also help my mind focus. The inciting itch for Tabula Scripta is trying to keep track of Doctor’s appointments, believe it or not. Now, I have normal calendars, and other sorts of things in the in-between, so it’s not strictly necessary, but I’ve always wanted something that was like Excel, but wasn’t an utter pain to use from my phone. A place to collect gas mileage records, where I am in certain comics, or other light-weight data-driven websites. Sometimes, I’d rather not have to set up a database to make something data-driven.

This is where the idea of spreadsheets comes in. They make a natural habitat for scratching out data sets, cleaning them, or doing ad-hoc data recording. Formulas even make it easy to do analysis over your data. Add in the idea of scriptable forms, and you can make capture simple and easy, and you can also make reports and charting possible, maybe even a daily summary of different things on your plate.

The thing is, there isn’t currently, to my knowledge, an actual Access-lite webapp. Not that I’ve done a lot of research into the type of software. I had done a lot of research into task-tracking and journaling software before Gills, and still have gotten a lot of mileage out of making my own. I’ve always had a bent towards making software tools for myself, I don’t see a reason to stop now.

So, with all of that rambling out of the way, what is my plan for Tabula Scripta?

Zeroth, Tabula Scripta is going to be a WebApp, designed to be accessible from Mobile Devices as well as desktop platforms. That doesn’t mean that it’ll be 100% easy to use from mobile phones at first, but I’ll try to avoid being mobile-phone hostile.

First, I want to get a functional beta of a spreadsheet editor working. Right now, the idea is for the client-side to send updates from the user as they come in, and for the server to send back all the affected cells. This means that formulas will mostly be calculated server-side, which allows me to keep the JavaScript code focused on presentation, rather than on modeling the intricacies of evaluating the formulas.

Alongside this, I’m developing a hierarchy of folder, scripts, forms and sheets. I might add notes in there as well, The intent of all of this is to make it easy to build a project in Tabula Scripta without having to fuss too hard with the outside world. Webapps whose state can be contained inside a single SQLite database + executable are easy to operate once one learns a reverse proxy.

After I get the notes and folders going, I plan on building out a scripting language, which I’m planning on calling TabulaScript. I anticipate it seeing significant evolution over the next 6 months, as I dogfood Tabula Sheets in different applications. For now, I’m planning on writing a stack-based language, for the sake of implementation simplicity, but long-term I think I’d like to aim for something a bit more conventional.

Why add my own scripting language on top of all of this? In part because I think I have a lot I learned from PISC that I’d like to try to apply, but also because having a custom scripting language will allow me to do some unique spreadsheet-focused things, like building in support for tracking spreadsheet dependencies throughout the language, so that I can detect data-cycles, even if you’re using heavily custom scripts. It remains to be see how practical that ends up being, but it’s at least a thought.

I also plan on making it possible to extend Tabula Scripta with your own formulas, built on this scripting language, rather than relying on only the set that it ships with. Below are some sketches of how it will probably look at first. Long term, I may try to make the scripting language a bit more conventional.

formula: [sum range] [
     0 ->'total
     per-cell: 'range [ getNumberFromCell 'total add ->'total ]
     'total ->'result
]

Tabula Script is also going to be at the heart of forms, I think this is where it will shine:

# A proposed form for quickly recording appointments:


UI: [
    markdown: "# Appointments" # Gets converted to HTML
    style: [myForm.css] # drops in a style tag into the HTML
    raw: [<em>Tabula Scripta is just ideas</em>]
    form: [ # Designed to make it easy to build up input forms
         use-table-layout # an HTML table to layout the inputs for
                          # this form for pretty alignment.
         # If an input has two arguments,
         # the first is the label,
         # and the second is the variable name that the for will expose on submit
         DateTimePicker: ["Appointment Date" "ApptDate"]
         # If it only has one argument, that is both the label *and* the name exposed on submit
         Text: ["Location"]
         LongText: ["Notes"]
    ]
]


Submit: [
 # A stack-based pipline for now
 "Appointments.sheet" get-sheet "B2" get-next-empty-row ->'anchor-point
 # Save row takes an first cell, saves the first value in that cell,
 # and then keeps saving cells along the current row.
 # There will probably be save-col and save-grid for similar other applications.
 'anchor-point [ 'ApptDate 'Location 'Notes ] save-row
]

I’ve not thought through every little thing just yet, but I hope these examples give a decent first impression.

Ultimately, the first version of Tabula Scripta is aimed at making it faster for software developers to execute on small-to-medium ideas, rather than aiming it 100% accessible to everyone. Because there are a lot of small ideas that are worth being able to build, from making it easy to set up a College Club recruitment website, to making it easy to record workouts, to making it easy to keep track of web-comics. I don’t think these ideas each necessarily need a dedicated webapp for them, but combined together, I think something like Tabula Scripta could make it easier to build more of those small projects.

Long term, I’d like to look into adding things like cron-jobs, making it easy to skin/style Tabula Scripta, some form of user-access control (but that is not a priority for version 1), and a way to have a notion of publicly visible things vs internal things that only the admin(s).

But, once I get sheets, scriptable forms (which I intend to double as reports), and extensible formulas, I’ll see where the itches guide next.

Nim: Scripting ease in a compiled language

Edit from November 2023

The main Nim developer has gone rather mask-off fascist. Given that, I can no longer recommend Nim as a language. Below is the original article, for posterity

Original Article

Nim went 1.0 around the beginning of October, 2019. I’ve taken the chance to dig into it for the past three weeks, and while I can’t call myself an Nim expert as of yet, I think I can offer a decently informed opinion.

I suppose I should add a bit of context: I’ve used Go as a side-project language since 2014, mostly for Gills, PISC, and the blog you’re reading right now. It’s definitely served me well, but I’ve been growing rather tired of some of it’s… choices. It’s not been terrible, but I’ve grown to miss generics and exceptions from C#, and Go just tends to be verbose in general.

That being said, there are some things I like about Go: It compiles on both Windows and Linux, it makes executable files that are easy to deploy (at least for my amateur sysadmin sensibilities), and it gets web servers spun up relatively quickly, all things considered.

For a while, I got around Go’s verbosity by using PISC and then Lua as scripting languages for the projects I had going in Go. I see myself using Lua as a scripting layer for Gills for quite some time to come, just due to how nice a search/tag driven CMS is.

But, when Nim announced their 1.0 status, I decided to do a few small projects to see if it was the language I wanted to make my side-project language of choice for the next 5-10 years, the way that Go has been my side-project language of choice (outside of games, where I’ve mostly used Lua) for the past 4 years.

Project Zero: Porting Jumplist

So, a project that I’ve landed on as a decent way to exercise a programming language without requiring the language to have a full web stack (which is a very heavy requirement) is to implement a directory jumplist manager. I first wrote one in bash that was an abomination of sed, awk, and bash that had slightly self-modifying code. The concept was to avoid calling into windows processes in git-bash, since that seemed to bring a non-trivial amount of lag.

When I wanted to bring the same jumping experience to cmd.exe as a result of using that a lot more at my new job, I opted to write a jumplist manager in Go and some wrapper batch scripts.

Porting this program to Nim was mostly a pleasant experience. I was able to remove about half my code, because Nim had a pre-existing insertion-ordered table, so I didn’t have to write my own insertion-ordered key-value store like I did in Go. The only grumble I had from this project that I had some difficulties figuring out some of the type errors I got at first. But, seeing half of the code I wrote disappear, and the code that I wrote being much smaller was really nice.

Project One: Porting kb_wiki

After I ported the Jumplist script, I had a pretty solid idea of my next target: The various instances of the kilobyte_wiki software. This was software I’d written in Erlang a while back. It was still running in tmux panes because I’d not figured out how to do proper erlang releases while using erlang.mk as my build system. (Side note: I recommend trying out rebar3 over erlang.mk, as with erlang.mk, you have debug a complicated makefile on top of the erlang tools. From what I’ve heard, rebar3 is more user-friendly than erlang.mk, which was difficult for this erlang and make newb to use). I also wanted to stand up an instance for tracking various articles and the best comments I’d found on Lobsters and Hacker News in a given month. In the process of porting kb_wiki from Erlang to Nim, I ended up finding a few things that kinda confused me at first glance, but that make sense in retrospect.

First, the standard library database modules only return database results as seq[string], roughly a List<string> if you’re from C#, or an ArrayList<string> if you’re from Java. This is a deliberate decision to optimize for the likely case where most of the data is going back into other strings in other parts of the application most of the time, and to try to help keep the database code from having to expose different sets of types for different databases, (as I gathered from talking Araq, the creator of Nim on IRC). While it’s not the choice I’d go with for a database application where I was heavily concerned with I/O performance, it’s certainly not a terrible choice for my more standard Sqlite+Web Server type stuff I’ve done a lot of lately.

Project One and a half: Understanding Jester

Also, though it currently (in October of 2019) not look the most kept up to date, Jester is definitely very nice to use as a web framework. Here are a few things that I ran across when porting wiki from Erlang to Nim:

You’ll want to make sure to read up on the settings: macro if you want to set what ports and hostnames your web server is binding to. I use it in kbwiki, if you want a small example.
cond expressions in the router that evaluate to false just won’t match, so you’ll get 404 errors, which can make debugging tricky. I recommend making them if statements if you’re trying to debug why a route isn’t matching, and then use your debugging method of choice to figure out what’s going on.
For now, Jester only properly supports HTML forms that use an enctype="multipart/form-data" in their HTTP posts. I don’t think this is a major problem, but it tripped me up when I was trying to forms to work on my website, so I figured I’d mention it here to save the next person making websites some time.
The Jester README current states that it requires a devel version of the Nim compiler, but I think at least the version in Nimble works fine with Nim 1.0. I am investigating this, but for now, it’s not something huge to worry about.
Jester is a bit underdocumented, which means that understanding certain parts will require reading it’s source. I didn’t have much difficulty there, and I’d like to help with the docs for it, as I have time. This paragraph is a first attempt at helping with them.

Cool things learned from two and a half projects

So, I had already learned from my first project in Nim that it was a lot more concise than Go, but it was in this second project where that truly became evident. See, Nim is the language I’ve had the easiest experience creating executables in. If you look at the code for kb_wiki, you’ll notice the main app, kbwiki.nim, and that it requires database.nim, kb_config.nim and views.nim. Unrelated to the main kbwiki app are mdtest.nim, todo.nim, and scraper.nim. They are all separate applications that can be compiled into executables using nim c mdtest, nim c todo or nim c scraper, and they can use other nim files.

But, because files are that easy to re-use, and executables are the easy to make, it makes code re-use much easier, at least in the small, than I’ve ever had it be until now. In Go, creating new executables involves a lot more ceremony, in terms of making sure that a new folder structure is created and such. In C#, unless you use LinqPad, you have either MSBuild/VS projects, or you have to create a whole folder and use then dotnet tool to create projects and explain to the C# compiler how to turn a set of source files into an executable. But, in Nim, you just write your file, install the right things via nimble, and then you can compile it into an executable.

Project Three: Scripts and utilities at work

This nimbleness of executable creation makes Nim the first complied language I’d use for general purpose command-line tools, where the value of writing 10 lines of code isn’t dominated by the process of setting up a folder structure and project files to get everything set up.

So, I’ve made some small utilities for my day job. An program for spitting out “banners”, aka the “-“ with a user-specified color in the background. A little command-line punch-clock for helping me keep track of what time I’m working (when working flex hours, it’s helpful to be able to punch in and out for personal tracking). A tool for copying these little tools from the directory they get developed in to the designated “My command-line tools live here to be on the $PATH” directory.

Project the Fourth: Scraping kb_wiki

Most recently, like I mentioned above, I wrote a scraper that can be pointed at instances of kbwiki and download their content. The motivation for the project was pretty simple: I had two instances of the Erlang version of kbwiki floating around, and I wanted to get the content hosted on them in Sqlite files.

The Erlang versions of the wiki use dets tables, because that’s what was easiest to get access to when I was writing the wiki in the first place. So, at first, I looked into trying to connect my Erlang code to SQLite. But, after digging into that for about 90 minutes, I was running into issues around gcc compiler versions on my linux box. Not wanting to spend until the wee hours of the morning trying to fight both Erlang, GCC, and Sqlite, I decided to switch from trying to dump the content into Sqlite files from Erlang, to writing a Nim scraper that basically gloms over the HTML, logs in as an admin, and then downloads all of the markdown for the articles. That took me just under two and a half hours (give or take) to finish, and then I was able to scrap down the wikis in a matter of seconds.

After that, I moved https://idea.junglecoder.com and https://drift.junglecoder.com from running on Erlang to running on Nim. And I didn’t have to fuss with certificates or the like, due to them running behind nginx. I also put up https://feed.junglecoder.com, where I’ve been keeping a feed of what seem to me to be notable comments, articles, and such.

Conclusions

So, having done 4.314 things in Nim in the space of just under a month, I’ve got to say that I quite like Nim. I don’t think it’s quite ready to be a programmer’s first language, a la Python or Javascript, mostly due to a lack of beginner friendly documenation. I also think it still has a lot of room to grow. But I’d like to try to help it grow. Jester is super handy for making websites with. I’ve discovered a whole new level of code-reusability, and the language is just so darn handy to build small/medium things with. I’d like to try to build something bigger with it, probably something off my idea backlog.

What 8 years of side projects has taught me

I’ve been a professional software developer for almost 8 years now. I’ve been paid to write a lot of software in those years. Far more interesting to me has been the recurring themes that have come up in my side-projects and in the software I’ve been personally compelled to write.

Lesson 0: Programming in a void is worthless

When I wanted to learn programming, I always had to come to the keyboard with a purpose. I couldn’t just sit down and start writing code, I had to have built an idea of where I was going.

For that reason, I’ve always used side-projects as a mean of learning programming languages. When I wanted to learn QBasic, there were a number of games, one based in space, another was a fantasy game. When I wanted to learn Envelop Basic, I attempted making a Yahtzee Clone, a Space Invaders Clone, a Hotel running game, and made a MicroGame based on the ones I’d seen videos of in WarioWare DIY.

When I wanted to learn C#, I went through a book, but I also, at the same time, worked on building a Scientific Calculator. (put a pin in that idea). When I wanted to learn Go, I wrote the CMS for the blog you’re reading right now. To pick up Lua, I used Love2D in several game jams. The only reason I have more than a passing familiarity with Erlang is because I used it for idea.junglecoder.com

Most times, when I’ve tried to learn a programming technology without a concrete goal to get something built, it is hard for me to maintain interest. That hasn’t kept me from trying out a lot of things in the past, but it’s the ones that allowed me to build useful or interesting things that have stuck with me the most. Right now, for side-projects, that list includes Go, Lua, Tcl, and Bash.

Lesson 1: Building a programming language is hard, but rewarding

Ever since I first started cutting my teeth on C#, the ideas of parsing have held a certain fascination to me. Like I said before, I started wanting to write a scientific calculator. But, because I was a new programmer, with no idea of what building a scientific calculator should look like, I did a lot of inventing things from first principles. It felt like a divine revelation when I reasoned out a add/multiply algorithm for parsing numbers. It also took me the better part of two weeks to puzzle it out.

I was so proud of that, in fact, that I copied that code into one of my work projects, a fact which really amuses me now that I know about the existance of Regex and Int.Parse().

Eventually, I worked out a very basic notion of doing recursive descent parsing, and some tree evaluation, so that, on a good day, I had a basic, but working, math expression evaluator.

Working on that calculator, however, set me on a course of wanting to understand how programming languages worked. In the process of wanting to understand them, I’ve looked over papers on compilers more than once, but never quite had the patience to actually write one out. In the process of wanting to make a programming language, I ended up writing two before PISC. One that was a “I want to write something in a night that is Turing Complete” langauge that was basically a bastard version of assembly. At the time, I called it SpearVM. I had intended it to be a compilation target for some higher level language, but it mostly just served as stepping stone for the next two projects.

The second one was a semester-long moonshot project where I wanted to try to make a visual programming language, using either Java Swing or JavaFX, inspired by Google’s Blockly environment. Unfortunately, I could not figure out nesting, so I giving the ideas I’d had in SpearVM a visual representation, and using that for my class assignment.

The combination of all of these experiences, and discovering the Factor language, set me thinking about trying to build a programming language that was stack-based, especially since parsing it seemed a far easier task than what I’d been trying to do until then. A couple late nights later, and I’d built out a prototype in Go.

I’ve had a number of co-workers impressed that I’ve written a scripting language. Thing is, it took me like 7 false starts to find a way to do it that made sense to me (and that was a stack-based language with almost 0 lexing). It’s only now, that I’m on the other side of that learning experience, that I’d feel comfortable approach writing a language with C-like syntax. PISC, as I’ve had time to work on it, has actually started to develop more things like parsing, lexing, and even compiling. In fact, I’ve got a small prototype of a langauge called Tinscript that isn’t nearly so post-fix oriented as PISC, though it’s still stack based.

And, PISC, to boot, is still what I’d consider easy-mode when it comes to developing a programming language. Factor, Poprc, or even a run-of-the mill C-like language all strike me as projects that take more tenacity to pull off.

Lesson 2: Organizing my thoughts is important, but tricky to figure out

If the early years of my programming side-projects often focused on how to build programming languages, and how to better understand computers, the more recent years have a had a much stronger focus on how to harness computers to augment my mind. A big focus here was for me to find ways to reduce the working set I needed in my mind at any given time. This has resulted in no less than 7 different systems for trying to help keep track of things.

A TCL application for launching programs I used on a regular basis.
A C# application for the same, but with a few more bells and whistles
Trying out Trello
Trying out various online outliners, ultimately being satisfied with none of them.
A months-long foray into trying to learn and apply Org-mode and Magit in Emacs, and ultimately giving up due to slowness on Windows, and the fact that my org-files kept getting too messy.
ideas.junglecoder.com, a place meant for me to shunt my stray thoughts to get them off my mind during the work day.
Another TCL application called PasteKit, which was designed to help me juggle all of the 4-6 digit numbers I was juggling at one of my jobs.
A C# version of PasteKit, that also had a customizeable list of launchers
.jumplist.sh
Bashmarks, but for CMD.exe

These are all approaches I’ve invested non-trivial amounts of time into over the past three years, trying to figure out a way to organize my thoughts as a software developer, but none of them lasted much longer than a month or so.

All of this came to a head during Thanksgiving weekend of 2018. My work at Greenshades often involved diving deep into tickets and opening a lot of SQL scripts in SSMS, and I had found no good way to organize them all. So, in a move that felt rather desperate at the time, I wrote a C# program that was a simple journal, but one that had a persistent search bar, and stored all of its entries in a SQLite database. And I used a simple tagging scheme for the entries, of marking them with things like @ticket65334, and displaying the most recent 5 notes.

It was finally a system that seemed to actually work for how I liked to think about things. The UI was a fairly simple 3-column layout. In the leftmost column, I had a “scratchpad” where I kept daily notes of what I’d worked on, in the middle I had my draft pad, and on the right I had the feed of notes, based on the search I’d done. I also had a separate screen dedicated to searching through all the notes that had previously been recorded.

There were several benefits to how this system worked:

It allowed me to forget things by putting notes under a different tags. That meant they wouldn’t show up on my focused feed, but that I could get back to them later.
It allowed me to regain context after getting interrupted much easier. - It gave me a virtual rubber duck. Since I often was trying to figure out issues by writing them to my teammates anyway, the journal gave me a very good first port of call when my stream of consciousness got blocked by an obstacle. This helped dramatically with keeping me off distracting websites like Hackernews or Reddit.
It allowed old information to fall out of relevance. One of the biggest problems with all the various tracking systems I’d used before, especially Trello and Org-mode, is that the system filled up, it was hard for old items to fall out of relevance without also being just a bit harder to access. Due to the nature of the feed, this system made it much more natural for information to fall off. And if I wanted it to stick around, I could just copy the relevant bits to a new note, which I often do.

All of this added up to me feeling like I’d found a missing piece of my mind. Almost like I’d created a REPL for my thought process.

Unfortunately, I don’t have that C# version any more. I do have a Go/Lua version, which is webapp based, though I still need to put some time into making the feedback-loop for it tighter, since my first versions weren’t quite as tightly focused on that, as much as they were focused on replicating the UI layout of the C# version. I’d argue that the tight feedback loop that the C# version had would be more important now, and I’ve slowly been working on adding it back.

The nice thing about the Go/Lua journal is that it’s far more flexible than the C# version, due to being able to write pages in Lua. Which means I’ll be able to use it for more things, and hopefully get that tight feedback loop back in place.

Lesson 3: Search is a great tool for debugging and flexible organization

Exhaustive string search of both code and notes has proven to be a surprisingly effective tool for understanding and cataloging large systems for me. To this end, Gills (my journaling software), Everything Search (search over the paths and file names on your laptop) and RipGrep have been extremely handy tools to have on hand. The nice thing about search as a tool is that it can be adapted into other things quite nicely. In fact, I would argue that fast search, both via Google, and via the tools I’d mentioned above, is one of the more influential changes we’ve seen in programming in the last 20 years.

Coda: Sticky ideas

8 years is a long time, and there are lot more ideas that I’d like to get into later. However, these are the ideas and things I’ve worked on that have proven to be surprisingly sticky. Perhaps they might help you, or give you some ideas of where to focus.

Published September 9th, 2019

The Kinship of Midnight Travel

There is a strange sort of kinship that I feel for other people I see when I’m traveling from one place to another after midnight. Part of it is that travel after everyone is supposed to be in bed means that I see things without all the people around, denuded of the hustle and bustle that often gives a place it’s charm or stress.

But then, spotting that lone trucker at 1 AM, or the lady working the airline check in counter starting at 3:30 AM, or the security guard at 4:30 AM, I know that I’m with people that also know what it means to see a place without it’s crowds. They also have seen the empty streets, the fog, the sprinklers running in the night. They know what it is to wake before the sun, or to chase the hours into the night.

Sometimes this happens in airports, today, it happened as I was driving from Carthage, MO to Joplin, MO. I saw a trucker pulling in off the highway onto one of the commercial districts in Joplin, and was reminded of all the times I’ve traveled in the dark, either at the end of long day in airports, or at the start of a long day on the road. And, I knew, that in some small way, that trucker also knew what it was like to travel at night.

For them, it likely has lost any of the romance or novelty that I still give it. But there’s still something to be said for that most remote of connections, if for no other reason that it’s just me and that other driver on the road, neither of us lost in the crowd, both wanting to reach a destination, yet still travelling in the night.

Thing is, I won’t ever meet that person. But we still shared a space, both members of the midnight traveling clan, a space that most people actively opt-out of. Many interesting bits of life are often found at the edges, where only a few people are paying attention, rather than in the rush of the crowds, where everyone already is, and has already seen them.

Factor: An impressive stack-based language environment

Recently the Factor programming language had a new 0.98 release, after a 4 year hiatus after the 0.97 release. Finding this on lobsters, I decided to take Factor for a spin again, after years of having left it mostly alone to work on PISC. I decided I wanted to try to build a (still slightly buggy) 4-function calculator, as I find that a good way to gauge how easy/hard it is to use a GUI system for small things, and as a way to gauge what Factor is like in general.

The (quite frankly) awesome

Probably by far the most impressive part of Factor that I’ve seen so far: Factor is the second language I’ve come across that I’d comfortably take on a deserted island without an internet connection (the other being Go with it’s godoc server). This is due to the fact that a) 95% of all Factor documentation I’ve seen is accessible from the included Help browser, and b) the Factor debugger/inspector can inspect itself in great detail. This is the cherry atop Factor: You don’t just get a REPL, you also get a really useful set of debugging tools (especially the object inspector), and can dig into any of the state in the system, and using the help tools, once you understand some basics, you stand a decent chance of being able to either correct the state, or diagnose the bug.

This sort of tooling is something that SmallTalk is famous for, but any time I’ve tried SmallTalk, I’ve usually bounced off of it because most SmallTalk implementations end up living in their own world, making it hard to bring some of my favorite programming tools with me, and making it hard to interact with the outside world (footnote 3). Factor, on the other hand, even though it has an image that can updated to keep state around, and has a really good debugger, still integrates well with the outside world.

Really well, in fact. It’s easy to scaffold a vocabulary USE: tools.scaffold "vocabulary-name" scaffold-vocabulary , and then run USE: editors.sublime (or emacs, or vim, or any of about 24 other fairly popular editors), and then run "vocabulary-name" edit and have it open the vocabulary in the editor in question. This allows you to open up any of the Factor source code in the editor of your choice. And when you’re ready to start using/testing the vocabulary, running "vocabulary-name" reload puts you right into compile-fixing mode, where you can fix the bug in question, and then reload the vocabulary. When I was working on a 4-function calculator using the Factor listener and Sublime Text, it was a really tight feedback loop.

The GUI framework has a pretty nifty sorta-FRP like approach to how to handles data binding in controls, allowing you to daisy chain models (data that can be updated over time) via arrows (models with a quotation they apply to a base model when it’s data changes).

Also, Factor has a mostly working dark-mode setup (there are some sections of the inspector that have poor contrast, but the listener is in a good state), if you dislike the default black-on-white color scheme (I found it difficult for some of the late night hacking I was doing). Run the following commands in the listener (assuming Factor 0.98)

USE: tools.scaffold 
scaffold-factor-boot-rc

Then edit the .factor-boot-rc to look like so:

USE: ui.theme.switching 

dark-mode

Then then run run-bootstrap-init save in the listener, and close and re-open the listener. Viola, Factor’s dark mode!

Consequences of (relative) obscurity

That being said, there are still a few rough edges with Factor. For one, even though most things are documented, and those that are not have easily accessible source, Factor itself has almost no Google presence, which means that you have to be comfortable digging though the included help docs a bit longer to sort things out. This played out in a few practical ways when I was working on the 4-function calculator. Until the end of the project, I missed that gadgets (Factor’s equivalent for controls in WinForms) were subclasses of rectangles, which meant that I could set their width and height by storing to the dim(ension) on a gadget:

"Test" <label> { 200 300 } >>dim "example of a 200x300 label" open-window

I ended up looking around the documentation about 4 times before I made that connection. This is the sort of question that would be on stack overflow for a more popular language.

For another, I seemed to be running across rendering or state glitches in the listener that could cause buttons to be mislabeled. I’m unsure what was leading to it, but it was distracting, to say the least. The other thing with Factor that became evident was the singled threaded nature of it’s runtime (as of currently). When first making a search in the help system, it would lock up, sometimes as long as 10-15 seconds, as it was indexing or loading in the various articles and such into memory. This limitation has kept me from digging into some languages (like Ocaml, though there are a bunch of other reasons, like iffy Windows support) in the past, especially when alternatives like Go and Erlang with their strong multi-threaded runtimes exist, but I think I’m willing to look past it in Factor for now, especially since I hear that there is a half decent IPC story for Factor processes.

The other consequence of all this was that it took me roughly 10-15 hours of noodling around with Factor, and writing an even smaller example GUI script to be able to get the linked calculator above done. I think I could build other new things with far less stumbling around, but it was a good reminder of how slow learning something new can be.

The other minor frustration I have with Factor is the fact that any file that does much work tends to accrete quite a few imports in its USING: line (the calculator I wrote has 23 imports across 4 lines, even though it’s only 100 lines of code). This is mostly due to a lot of various Factor systems being split into quite a few vocabularies. I could see this being helpful with compilation

Comparisons to PISC

Factor was a big inspiration for Position Independent Source Code (PISC), especially since it gave me the idea of code quotations some basic ideas for a few stack shuffling operations and some combinators (like when or bi). Factor and PISC have diverged a decent amount, though Factor is far and away further down the general path I’d been wanting to take PISC in terms of documentation and interactivity. Revisiting it after spending 3 years of assorted free time working with PISC demonstrated that in a lot of ways.

Factor has a much higher level of introspection. When making the calculator I only had to think in Factor, rather than thinking in both PISC and Go (which is common when writing PISC). I also didn’t have to rebuild many building blocks to get where I was going, like I would in PISC, as Slava and the other maintainers have done a lot of the foundational work already. Revisiting Factor after having written so much PISC did make understanding the stack a lot easier. I imagine that stack management will be most people’s struggle with Factor. I’ve found that implementing a stack language helped me there, but for some people, they may just have to do some exercises with it.

Conclusions

I do anticipate building more side-projects in Factor, as I’ve yet to try out the web framework, and I have a couple of ideas for websites in the back of my mind. There’s a lot of vocabularies that come with the base Factor download. I get the feeling that I may be adding some to it in the future, though time will tell.

Two Android Tips

Two random Android related tips:

If you’re developing apps on less than common Android phones via Andriod Studio, like the One Plus One (my device), then using the ADB network bridge will save you having to figure out which set of drivers you’d need to actually be able to connect to the phone via USB. It is cautioned against if you’re on an untrusted network, and at least the One Plus One has you confirm that you want to create a debugging connection, but it does seem a bit handy.
If you’re using a Microsoft Bluetooth 5000 or 6000 keyboard and can’t quite figure out how to get it to pair, try long-pressing on the Bluetooth/Pair button. I was trying to use my Bluetooth keyboard after a long hiatus of not using it, and stumbling across this review helped me figure that out, when manuals and the like were not very easy to find.

Published May 10th, 2017

PISC now has a Github Mirror

A while ago, I decided to self-host the PISC repository using Fossil, mostly so that PISC could be easily self-hosted on hardware that I own, rather than relying on Github. Until now, this meant that to play with PISC, you had to either install fossil (which isn’t hard, but is an extra step), or content yourself to a tarball/zip file.

As of this morning, there is now a Github based mirror repository. While I don’t intend to do my main development and documentation there, I will accept and work with issues and/or PRs there (such as I have time to do).

Published April 17th, 2017

My Programming Journey Part 2: Calculators led to Parsing led to Programming Languages

Every programmer has a favorite toy problem or smallish program idea that they like to use to try out new tools. For quick testing of GUI tools, mine is a 4-function calculator. This fascination started at an early age, when I was programming QBASIC at the age of 14. I wrote one in 15 lines of code that prompted for the first number, operation and second number. When I moved on to Envelop Basic, I created a GUI calculator with three text boxes, two inputs and one output, with a button for each of the 4 operations.

Then, as my family was traveling from Kansas to Florida in 2011, I was introduced to Petzold’s CODE, and advised that C# would be a good language to pick up for finding jobs. One of my key discoveries working in BASIC during my teenage years was that I had to have a project in mind if I was going to get any programming really done. So I decided that to learn C#, I would build as much of a scientific calculator as I could figure out.

The days I spent pondering this problem were heady and inspirational. My only other general knowledge of the field came from Coders at Work and Head First C#, 2nd Edition. My Googling skills were non-existent. Working on that scientific calculator, I independently discovered how to write integer and decimal (digits and decimal point) parsers, after pondering the problem for a few days. At first, I attempted to convert from math strings into my own assembly-esqe op-code language, but I struggled to handle varying length arguments as I seperated the code from the data into separate arrays.

As I gained comfort with objects and references via my weekly exercises in Head First C#, I also conceived the bare basics of recursive descent parsing and tree-walking evaluation, which allowed me to handle parenthesis, without actually reading a reference on either. At one point, I actually had a calculator that could take strings containing numbers with decimal points, the 4 math operators, and parenthesis, and return a correct result. Then I decided to try and add Square Root.

Now, at the time, my programming research skills were still anemic, and I was still rather ignorant, so rather than using the now-obvious Math.Sqrt in C#, I began research on how to implement my own. This was about where that project fizzled out, as I ended getting my first software development internship in C# and then had my hands more than full trying to keep up with it. But even though it was never completely finished, it left me with two things: a fascination for parsing, and a attitude of figuring things out, no matter how initially confusing.

Since then, I’ve had multiple attempts at creating my own programming language, usually getting stuck somewhere in the processor of trying to define a grammar and write a parser. Side projects can be like that. Part of the problem is that I didn’t want to write a lisp, but didn’t have the time/concentration between work and school to properly hand-write a recursive descent parser for anything else I was interested in at the time. Looking back, I probably should have embraced ANTLR or bison, but that’s another story for another time.

At one point in 2014 or so, I finally decided that I wanted to just make something Turing Complete, so that I had finally done it. The end result was a line-oriented, half inspired by CIL, and half crazed VM-esqe interpreted thing that only spoke in a stack of numbers, and a buffer crafted out of a couple late nights with Go on my laptop. A Fibbionaci and looping program later, and I was satisfied to have finally done something that could be considered Turing Complete.

A semester or so later I ended up porting the same language into a Java version. My original plan had been to build my own version of Blockly or Scratch’s UI, and build an AST evaluator below it, but dynamically nesting UI in Swing and JavaFX was something I couldn’t figure out inside that semester. I do believe it is possible, but it is likely far more effort than it’s worth.

Not long after this, I ran across Factor. After fiddling with it for a while, I began to realize that I could probably manage to parse quotations and word definitions, and wrote up a simple as possible proof of concept in Go. I think one late night finished a basic parser and interpreter that was able to handle booleans, numbers, word definitions, and quotations and some basic operations on them. I didn’t have a name for a while, but eventually landed on a pun of the idea of Position Independent Code: Position Independent Source Code, aka PISC.

PISC has served a few purposes for me. Firstly, it served to cross “Implemented reasonably complete programming language” off my bucket list. Secondly, since the basic implementation was completed so quickly, I’ve decided to use it as a means of learning all of the non-parsing things related to building a programming language. So far, that’s been things like building a standard library and writing a syntax highlighter for Sublime Text 3. Going forward, it includes things like figuring out how to best distribute it while using fossil, finding a good niche for PISC. I’ve also had the opportunity to spend some time trying to optimize it, though there is a lot of performance that can still be gained.

One of the surprising things about PISC is how easy it has been to work on. Building the various bits and pieces of it has rarely taken more than 10 hours per bit. Stack based programming languages are often overlooked, backwards flowing nature their because. But because they are so easy to parse, and rather syntactically flexible, they lend themselves to very easy expansion. (And PISC isn’t even properly meta-syntactic yet). The other reason for this is that I’ve gone down the road of creating a dynamically typed programming language with a minimal number of types, and little to no type-level reasoning, at least yet. PISC only has 9 types at the moment: Integers, Doubles, Strings, Vectors, Dictionaries, Symbols, Quotations (which do live on the stack), and Native Go Functions.

PISC will be one of my side projects for some time to come, as there is still a lot to learn and design with it. I think stack based languages might be my new programming toy problem.

Published March 19th

Late Night Programming

I’ve decided to tackle it
It’s going to get done
Doesn’t matter if I’m fit
Or if it should be fun

Sleep becomes an afterthought
Thoughts narrow into bytes
Solutions, researched and sought
slowly form neath flickery lights

Abstractions are conceived
Ideas are slowly sprung
One works this time, I’m relieved
Unlike the first 10 things I’d flung

But as I go more bugs crawl out
My understanding is incomplete
Undeterred, I consult cout
To render those bugs concrete

And through the night errors retreat
Further through this code we go
“Please work” I oft entreat
“I need this for Dr. Guo”

And finally the data’s stored and clear
And knowing I’ve done it, relief
Tho hours blurred to seconds mere
My work now strengthens my belief
That I can solve problems

So it’s done for now
Still sits in my head
But as sleepy as a cow
I go to bed

JS: DOM Attributes are lower-cased, and getAttribute is case insensitive

A short reminder for those of us writing JS without libraries:

DOM attributes are a little strange from JS. When you iterate over them using the attributes property, all the names come back completely lower cased. When you fetch them using getAttribute(name), the attribute name argument is completely case insensitive.

Short Code Example:

(Code shown below for anyone that just wants to see it)

<div id="my-id" data-ThisIsInfo="Hello, Internet">
  <script type="text/javascript" >
  var elem = document.getElementById("my-id");
for (var i = 0; i < elem.attributes.length; i++) {
  var attrib = elem.attributes[i];
  if (attrib.specified && attrib.name.indexOf("data-") === 0) {
    elem.innerText += attrib.name + "='" + attrib.value + "'";
    console.log(attrib);
  }
}

elem.innerText += elem.getAttribute("data-tHISiSiNFO");

  </script>
  Debug:
</div>

Life in PNG: Moving and moving and moving

I had the opportunity to visit my parents in Papua New Guinea this summer, and it was an awesome trip. It gave me a chance to clear my head and get away from the constant stream of information and distraction that is so baked into life as an American technologist. It also gave me a chance to get a perspective on what missionary life is like.

One example of life overseas being just straight hard work is the sheer amount of moving that is involved. First, you have to pack up your entire American life. Every. Single. Thing. All your books, clothes, cars, shoes, plates, food and everything has to be dealt with in some fashion.

Then you travel halfway across the world, often in economy seats, as you’re already spending $2500+ a person just to get where you’re going. You take as much luggage as you can manage. The overall trip takes a bare minimum of two days, where literally half of that time is spent sitting in an airplane, trying to catch some sleep while the plane engines are roaring and it’s the wrong time of day for your body. Airports like DFW are entertaining, amusing places to be when you’re not stuck in a line, which helps.

Finally, after overnighting in a hotel and clearing customs 2-4 times in different countries, you end up in PNG. But you just finished the easy part.

Now you have to go and do third world shopping for your first few days in the country. The total selection at all the stores in town is roughly equivalent to the first half of 3 aisles at Lowe’s and a Save-a-lot. If, like my family, you’re on a gluten free diet, you options just went down to about a 1/5 of what they would have been.

Then you take what’s left of your day and make beds, pull dishes out of storage, and make supper, which has to be cooked from scratch <a href=”#note-1” id=”ref-1”>1</a>. Exhausted after traveling and shopping and setting up house and cooking, you fade onto your bed and try to sleep in the humid night. You’re under a fan because it’s way too humid to try to sleep with sticky-tepid air.

Then, after a week in town to gather supplies and run town-based errands, you begin packing up your house there and getting ready to move out to the village. In my case, the village<a href=”#note-2” id=”ref-2”>2</a> is only 1.5 hours away, but because all seven Owen family members are in country, we have to make two trips. Thankfully, this means that nobody is sitting atop each other. Life in the village was pretty good, but a bit awkward at times, as my Pidgin was rusty, and I didn’t know what to say to different people. Thankfully, people were glad to see me even when I was more silent than I had been as a child.

Another week later, we ended up taking a trip to an SIL<a href=”#note-3” id=”ref-3”>3</a> mission station called Ukarumpa for some rest and relaxtion. Villages are noisy, busy places with people that have all kinds of sleep schedules, which may or may not coincide with yours. Sometimes it helps to get away from it all and recuperate. Ukarumpa has a much more structured lifestyle, with a nightly noise curfew. This helps with catching up on sleep. It is also at a much higher altitude than Uria, my parents village, which means that the temperature hovers between 40F and 70F, like natural air conditioning.

However, that means more moving. Ukarumpa is a five hour drive from the village, which means that the two trips that get you out to the village aren’t an option. So, you work out what’s essential: Food, pillows, bedding (not mattresses, thankfully), books, computers, the 10-odd books that make up that week’s worth of school, dog-food and Bananagrams <a href=”#note-4” id=”ref-4”>4</a>. This is all loaded into the back half of the truck, on 1/3 of the floor of the front of the truck <a href=”#note-5” id=”ref-5”>5</a>, and the laps of all the non-driver(s).

The end result of this packing and evaluation process is that all seven of us are in two rows of seats. In the front is (left-to-right) me, Mother, sitting in the middle with the stick-shifters and then Dad with the steering wheel and pedals. Thankfully it’s mostly comfortable up front, as each passenger has enough seat for both sides of their bottom. In the back is Abigail, with 1/4 of a seat, Hannah and Josie in between, and then Samuel with 3/4 of a seat holding our dog, Meggie. The back is hip-bone to hip-bone. Meggie, though she does travel well, is still fidgety and fussy on this trip, which makes for interesting commentary from my brother.

So, with the truck packed to the gills like the Clampetts, we go on the 5 hour trip from the village to Ukarumpa. After about 4 days up there, reminiscing and enjoying the cool, we start to go in reverse, traveling to the village again, except that this time I sit/swim in the back with the cargo. Then we spend a weekend in the village, saying goodbye to people who may never see us again. And then back into town for a week, with all the same shopping and unpacking that had happened the time before, and then we have to pack up and to redo that international travel.

Does it sound exhausting? Bear in mind, this is all before any translation or ministry can take place. It’s also top of whatever attacks Satan has lined up for you in a given time, which are never absent to those following God’s call.

My parents are as tough as diamond blades cutting through titanium. I thank God for the strength and grace that he gives to them as they follow his call, and live this difficult, transient life. It’s a blessing and an honor to be their son and carry their story in the USA.

How beautiful on the mountain are the feet of those who bring good news. - Isaiah 52:7

Notes

1:2:3:4:

Bible Translators never travel light. - Mike Sweeney

Why I dislike the Java Ecosystem

I am a programming polyglot. While I work by day in C#, I have wandered far and wide in my own time. Each language and surrounding ecosystem I’ve tried has a different strengths and weaknesses*, but one stands out as disappointing to work with overall: Java.

Java, the language, is a little bit on the verbose side, and a little split-brained in how it treats primitives vs classes, but it’s very far north of tolerable, especially as generics and such have been added over the years. Java, the ecosystem, on the other hand, is a case study in making things more complicated and painful than it needs to be, especially on Windows.

The first experience that students at FGCU go through when learning to work with Java is setting up and installing one of the two well known IDEs for the language: Eclipse or NetBeans. It seems that of the two, Eclipse is by far the better known and often used, but Eclipse has many little issues that come up. Setting up Eclipse for the first time is a drawn-out, arcane (especially for new programmers) experience.

First, you have to install the JDK. Thankfully it has a Windows installer that is kept up to date, but you’d better remember where it put the JAVA_HOME. Then, you have to download Eclipse as a .zip, extract it, and remember where you put it! Here’s one of my biggest complaints about Eclipse: It doesn’t come with a Windows installer. While it does have Windows downloads, without offering a Windows Installer, it’s immediately that much less pleasant to work with.

Let me be clear, I have no problem with providing a portable download for users to put where they see fit, and if a program is a single, self contained binary it doesn’t likely need an installer. But let us be clear: IDEs are some of the most complicated user-facing software in existence! The least that could be provided is a nice means of getting from 0 to an editor.

Now, if it was just Eclipse, that would be one with. The Visual Studio installer often takes 2-5 hours do get itself setup, but it at least covers all the bases needed. But the Java ecosystem has other tools that are needlessly difficult to work with. Apache Spark is one that stands out as being decently difficult for very similar reasons.

Spark, like Eclipse, doesn’t come with a Windows Installer and can be very fiddly to get working. I’m using it for a bit of research (as needed by my professor), out of 3 people trying to get it to work on Windows, we all have ended up installing it on Virtual Machines so we could move on with our research and life.

This isn’t to say that Java can’t be used to make elegant tools (like IntelliJ), but much of the culture around Java discourages elegance in favor of bureaucracy, all while not providing decent support to the OS that provides the majority of it’s users.

Ideas for LD36

So I had a bunch of ideas for Ludum Dare 36, as I tried to come up with an idea for each idea. As luck would have it, the theme was the one that I didn’t plan for, but these ideas could still be made to work. (Edit: I wasn’t able to finish anything for LD 36 due to life getting in the way)

Enjoy!

Ideas for each possible theme:

A different kind of Combat: Build a better color scheme to win in cage/WWE fights
Alchemy: Top-down collect-a-thon for the ULTIMATE POTION
Beyond the Wall: You try to leave the town that no one leaves. Ever. And there isn’t a way out.
Controlling Many: Change your color (red,blue,yellow) to get people to follow you and solve physics puzzles
Day and Night: During the day you can move, at night try to survive the challenge of where you are to reap the reward.
Don’t Kill Anything: You have crazy neighbors and you’re on the BRINK of killing them. Use QTE’s?
Going Deeper: Using a trusty hookshot, find and slay the Balrog of Morgoth
Hunted: You are a fox. The hounds are coming. Jump and dig your way to vitctory
Inconvenient Superpowers: You are made of rock. Try not to break anything during your daily life.
Isolation: Abducted by Aliens, try to lose your mind in the most harmless way possible.
Keep It Alive: The plant giving you O2 needs water and light, both in short supply down here
Level Changes While You’re Playing: Dash from safe chamber to save chamber as water alternates with lava in filling the screen
Magic Gone Wrong: You’re a newt trying to become human again.
One Enemy: The Black night REALLY wants to keep you from passing him on the road
One Room: Chase that darn button around the room, but you have to deal with the fact that button doesn’t want to be pushed
Plan then Execute: You are a jet-lagged sleepwalker with a timed safe that you need to get into.
Power Source: Using junctions, complete the wiring
Start Small: Metriodvania
Upgrade: Platformer where every time you land, you get an extra jump/boost that needs to be used on the next jump.

Published August 24th, 2016

Adventures in using Nginx as a reverse proxy

Lately I’ve been trying to learn how to run more than one web application on a web server. I understand this often involves using nginx as a reverse proxy. I’ve been interested in rewriting my blog in Elixir/Phoenix, but I want to keep the current blog up while I’m making the changes. So I’ve been stumbling through nginx reverse proxy tutorials. I will be taking notes into this post about various things that I’ve learned about how to set up nginx as a reverse proxy for golang applications like this blog.

Currently my config is below:

Some notes:

I needed proxy_cookie_domain in order to be able to authenthicate into the blog correctly. That was about 2 hours of work to figure out.
It seems that nginx can also serve as a proxy to other webservers, not just locally.

That’s all for now.

server {
        listen 443;
        server_name junglecoder.com;

        ssl on;
        ssl_certificate /etc/letsencrypt/live/junglecoder.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/junglecoder.com/privkey.pem;


        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
        ssl_prefer_server_ciphers on;
        ssl_session_timeout 5m;
        ssl_session_cache shared:SSL:50m;
        # ssl_stapling on;
        # ssl_stapling_verify on;


        location / {
                proxy_cookie_domain localhost junglecoder.com;
                proxy_pass http://localhost:8080;
        }
}

PowerShell: Organizing Desktop Detrius into Dated Folders

I was reading an article written by a linux sysadmin about the trials of good file organization. After trying various hierarchical schemes, he settled on an interesting one: Storing files by Date, in folders. I’ve also just learned about Engineering Notebooks, where ideas are logged for a company, and dated by page.

I decided to try something similar with a pile of documents on my current laptop. Hence, the following PowerShell script, run over several file extensions typs (see *.jpg).

    
    # 2015 Andrew Owen
    # Permission is given for any use, no warranty provided or implied.

    # 
    # I use the abbreviations that ship with PowerShell on Windows 8.1, 
    # to make it easy to show how concise PowerShell can be, 
    # compareble to Perl, Tcl or ruby
    #

    # ls    -> Get-ChildItem
    # sort  -> Sort-Object
    # gu    -> Get-Unique
    # mkdir -> New-Item -ItemType Directory
    # mv    -> Move-Item

    # Move to your desktop
    cd "C:\Users\$([Environment]::UserName)\Desktop"

    # Create folders for each of the files of w/ the extension below.
    ls *.jpg | % {$_.LastWriteTime.ToString('M-yyyy')} | sort | gu | % { 
        mkdir ".\datedNoteBook\$_" 
    }

    # Move the files to the folders that they belong to.
    ls *.jpg | % {
        $subfolder = $_.LastWriteTime.ToString('M-yyyy');
        mv $_.FullName ".\datedNoteBook\$subfolder\$($_.Name)";
    }

Make sure you're able to run timer based services on demand

My job has had me working on service applications that run on a timed interval. One aspect of debugging these services that was slowing me down was waiting for the service to get triggered. For the longest time, I compensated by having the service be in on a 10 second timeout, since it didn’t take very long to run a sync. This still slowed me down though, and it made it easier to get sidetracked while waiting for the service to fire.

The solution didn’t strike me for a good 3 months: Add a way to trigger the service on demand. It is a very obvious solution in retrospect. And it makes a lot of difference when debugging, as I can fire off the service as much as is needed to test for different states.

A .NET breakpoint that only breaks if a debugger is attached

This is a helpful little function that I wrote when I found out about the [System.Diagnostics.Debugger.IsAttached](https://msdn.microsoft.com/en-us/library/system.diagnostics.debugger.isattached(v=vs.110\).aspx) property. I found it useful for debugging a synchronization service that I was working on at work.

What other little utilities like this have you made?

public static class utils {
    public static void DebugIf(Func<bool> toCheck)
    {
        if (System.Diagnostics.Debugger.IsAttached && toCheck())
        {
            System.Diagnostics.Debugger.Break();
        }
    }
}

Usage:

using System;

class App {
    public void Main(string[] args) {
        // Fires the debugger if one is attached,
        // and if we passed a flag asking for debugging
        utils.DebugIf(() => args.Length > 0 
                            && args[0] == "--debug");
    }    
}

Academic Portfolio

EDIT: These are rather old compared to what I’ve had time to do since.

Below are some links to some of the more interesting programs that I’ve done for school. None of these programs necessarily represents my best work, but you find might them entertaining or educational. This list will get updated as I add more of my learning-experience programs online.

Riemman Sums Calculator

This program calculates a Riemann Sum based on user inputted expression. I collaborated with a friend on this one, and with some help from our Calc 1 professer, we presentated it at the FGCU ASPIRE conference in 2013. My favorite bits about this project included the oppurtunity to work with Reactive UI. Here is the code on github.

PyAnalogClock

A late night (plus some tinkering) project that applies basic trig to create an analog clock using python and the tkInter UI. It was for my COP-1000 class at then Edison State College. Code on github

CKEditor setData failing - Documenting the Arbitrary

UPDATE:

The lead CKEditor developer responded to this blog post via a ticket.

The post as it was before:

When programming, the most difficult parts are the arbitrary ones. These are things you don’t know and couldn’t reason your way with what you already know. When you are starting in programming, this is most things. “Simple” is only when you already know how it works. It gets better the longer you program, but “simple” still takes a lot of time.

A good example was a bug that I recently helped a colleague hunt down. Frank, my co-worker, was struggling to reproduce a bug related to the CKEditor library. Below is the code that was failing, and relevant documentation. The bug: Sometimes, but not very often, setting the data will fail to actually populate the editor with the html in data.

       gCurrentEditorObject.setData(data);

The fix happened to be twofold:


  if (CKEDITOR.status == 'loaded') {
      gCurrentEditorObject.setData(data);
  } else {
      CKEDITOR.on("instanceReady", function (event) {
          gCurrentEditorObject.setData(data);
      });
  }

The first part that Frank and I discovered was the code inside the else clause. Frank worked out the code if statement over the course of a couple hours the following day. The basic logic is “simple”. According to my coworker:

Pretty straightforward. There was a minor timing issue where in some cases if the object wasn’t ready, calling setData on the gCurrentEditorObject did absolutely nothing. Adding the wrapper to avoid any execution before instanceReady state prevented the setData call from occurring prematurely. To avoid breaking the timing of the instances where everything was in order, there is a simple check for status == 'loaded'. Simply put, if the editor object is loaded, then carry on, otherwise wait till it is loaded and THEN carry on.

I hope this helps anyone struggling with the lesser known parts of the CKEditor API. It does make me wonder why the editor code didn’t have this guard in place around setData already. Delaying the loading of a javascriptAPI isn’t unheard of, but it’d be nice for the function name to somehow communicate this. But, if you work with the CKEditor, this is just something you need to know. Heh.

Published January 26th, 2015

International Bible Day

I found out this evening that today was the International Day of the Bible. In celebration, I bring a verse and a call.

Verse

How beautiful on the mountains are the feet of those who bring good news, who proclaim peace, who bring good tidings, who proclaim salvation, who say to Zion, “Your God reigns!”

Isaiah 52:7

Call

Todd and Angela Owen (my parents) are currently in the process of raising support to return to Bible translation in Papua New Guinea, among the Somau Garia people, after 7 years of ministering in the US.

Bible translation is hard, dangerous work in the third world. I’ve had my life at risk a number of times, just being along for the ride. I don’t regret it in the least. It would be awesome for my parents to be able to pour their energy and wisdom into this work of God.

For storing treasure in heaven, advancing the gospel, and equipping beautiful feet to share the good news, please pray for and donate to my parents. I know no one better equipped for this calling.

Published November 24th, 2014

Meta: Bugs and Suggestions

Thank you for reading my blog. This website is currently done on my spare time, often during late nights (like much software). Because of this, there’s occasional bugs and ugly pages. I’d love to know when that happens, so that your reading experience isn’t sullied by it.

If you notice a bug or horrendous UI, it’d be awesome to either create an issue on this site’s github repository, or leave a comment below.

Counting lines of C# using Powershell

I recently wanted to see how many lines of code were in a particular C# project. Google lead me to the following article: http://www.limilabs.com/blog/source-lines-of-code-count-using-powershell

I refined that script to exclude most of the designer files that Visual Studio generates via UI tools. The results are below in a gist.

Published November 8th, 2014

Performance Gotchas in .Net 2 - Regex timeouts

Every programming framework has certain corner cases suck the performance out of an application. The .Net Framework is no exception. I’ve discovered a few in my work with C#, and blog about them as I find time.

I love using regex for search/replace/text manipulation tasks in programming. You don’t have to go to full perl mode for the awesomeness that is Regex.Replace("(\w+)-(\d+)", "$2-$1");. This awesomeness is counter-balanced with the risk of catastrophic backtracking when using the non-regular varieties of regex, like Perl, Javascript and, most pressing for me, .NET.

Catastophic backtracking takes exponential time, leading to long page load times, unresponsive applications and denial of service attacks on websites that use broken regexes. Jeff Atwood covered some of the bare basics of horrendous regex performance here, ending with a wish for a way to keep regular expressions from going full into a full ReDoS on your server. Some years later in .NET 4.5, his wish has been granted. C# and VB.NET now allow you to specify a TimeSpan that denotes how long a regular expression is allowed to take before giving up.

Below is a C# snippet based on Atwood’s code that can be run in LINQpad to illustrate the timeout in action:

string pattern = "(x+x+)+y";
RegexOptions options = RegexOptions.None;
Regex re = new Regex(pattern, options, TimeSpan.FromMilliseconds(1000));

string success = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxy";
string failure = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";

re.Match(success).Dump();
re.Match(failure).Dump();

Even though this is no excuse to keep from writing proper regular expressions, I like that it creates another layer of defense in depth against denial of service attacks based on the clever intern’s bad regex from last summer.

Life as a Pedestrian: My First Few Days in Bellingham

Note: This is a piece that I wrote towards the beginning of my time in Bellingham. I plan another article soon as I am on the other side of my internship

The world is a bigger place when you have to walk everywhere. This is my profound discovery of these first few days in Bellingham. Walking is one thing that I’ve done a lot over these past few days. It gives you a different perspective on life. I’m thankful to the many drivers that have stopped for me to cross the road these past few days.

The apartment that I’m sharing with Stephen Morrison is nicely situated for all the walking that I’ll likely be doing this summer. It’s under two miles from both my job at Logos and from Dollar Tree. A Fred Myer’s only five minutes away rounds out the supply of necessities very nicely.

Walking also gives time to enjoy the hilliness and beauty that is around Washington. The Northwest is beautiful country. There are trees everywhere, and Bellingham is situated on a lake. I have yet to see everything around here, but even walking around the downtown and near my apartment shows a lot of beautiful flora and businesses.

Coffee shops are abundant in Bellingham. My favorite hangout so far has been Wood’s coffee shop, since I’m able to get free wifi when there. My favorite named one would have to be the The Black Drop simply for their logo:

Being required to walk in order to get Internet is also a productivity boon. Once you see this post, I’ll have stopped by the Wood’s Coffee Shop to get it uploaded to my server, but I wrote it at my apartment. Due to a scheduling mix up on my part, I am without Internet access at the apartment for at least another two days. Predictably, I can get more done when using my computer, such as writing this blog article and finishing the greater part of my blog software.

Walking also gives a different perspective on groceries. Getting more than one gallon of milk? Not if you want to get anything else. That trip to the dollar store 1.6 miles away doubles as an upper body endurance workout. You want to buy groceries, because it means that you don’t have to leave the house every time you want to eat something.

Overall, I think all this walking will be good for me. Time will tell…

Published Summer of 2014

My programming journey: Part 1 - QBasic

One of the things that I’ve been blessed with is the opportunity to start programming from a young age. What follows is the start of that story, what abstractions and tools I picked up then, and what crazy things I was trying to do with them (more than I knew what I was getting into, usually). This post covers the 12-14 year old me learning ropes of programming (as remembered 8 years hence).

The Olden times: QBasic and Boy Scout Magazines

I started in programming for the same reason as many others: I had video game ideas that I wanted to make on the computer. My father, a Bible translator, set me up with QBASIC. My textbook was a pile of Boy Scout magazines that I found in the kid’s room at the Pioneer Bible Translators offices.

I started by copying a relatively short program out that described a dog sled race. It was around 20 lines of code, but it had more going on than I could easily understand. IF statements seemed natural enough, but the FOR loop syntax was tricky to copy correctly. I was able to get it working however, and found that I understood enough to try my own ideas.

From this sprung my two great tools for working in QBasic: IF and GOTO. From those I formed my loops, as the QBasic FOR loop syntax escaped my 12 year old mind. IF and GOTO I understood, FOR I 1 TO 10 ... Next and While ... Wend, not so much. I typed in line numbers so that I could mimic the Boy Scout magazine coding style, and learned how 20 INPUT NAME$ was different than 20 INPUT NAME (one is a string, the other is a number).

One of my first programs was a Samwise Gamgee vs a Goblin RPG game round. “Choose 1) for Stab or 2) Sam’s Fury” was the prompt. Damage was calculated by multiplying your integer choice by 10, (10 damage for a dagger, 20 for a sword). Upon dealing 40 damage to the goblin, you won the game, provided that you survived his dicey assault.

Eager to share my accomplishment, I recruited my brother to test the game. It didn’t take him long to chose attack option 20 instead of 1 or 2. That attack hit for 200 damage, killing my 40 hit point orc in one blow. Thus I learned the importance of validating user inputs. It took me 45 minutes to patch that, if I recall correctly).

My most finished program in QBasic was a simple 15 line calculator application that didn’t have a good user experience. It didn’t really explain itself, but could apply the 4 basic operations too a pair of numbers. That was the one that I could show to guests as an example of my computing prowess. The fact that said program ran in a CMD prompt was +5 to its coolness.

I then proceeded straight into biting more than I could chew. My game concept was "Final Frontier", a text-based based trading and combat game set in outer space. It was going to have 3 different spaceships, 4 trading ports, secret pirate bases, asteroid fields where combat took place and colored text!

Stymied by the SUBROUTINE

As I was learning QBasic, I discovered this magical thing called a SUBROUTINE that could be used to split work across several windows (it looked like several files). SUBROUTINEs kept me from progressing on Final Frontier, mainly by being hard to understand.

My first mistake with SUBROUTINEs was assuming that reaching the end of one would terminate the program. That lead to this exchange:

Me: “Dad, why does the program keep going after the end of the code?”

Dad: “The subroutine goes back to where you started it from when it’s finished.”

Me: “Oh, so that’s what’s going on…”

My reasoning, though flawed, makes sense in retrospect:

Programs stop when it got the end of the code
The subroutine has an end to the code
Therefore the program should end at the end of a subroutine

One must understand that I had no concept of the stack, or any solid understanding of function calls.

But, this newly corrected knowledge in hand, I attempted to use SUBROUTINEs to break up Final Frontier into pieces. The first one worked well for my main menu and intro sequence (I used terminal colors to make things look cool. Red text was rocking awesome for me in 2007!). But using SUBROUTINEs broke down as soon as I shifted to the code for the market place (my second SUBROUTINE). The intro sequence stored the ship choice of the player into a variable, but the marketing subroutine couldn’t see that variable.

For that matter, every time I came back into the marketing SUBROUTINE, I couldn’t see the variables from the other subroutines. That mean that I couldn’t tell where I had come from, or what ship I was using! How was I too solve this? The problem proved to be too much for my attention span and resources at the time, especially since I didn’t know how to Google very well. I did spend several hours trying to get it to work, trawling the help documents for information. This didn’t help much, as the documentation was opaque to non-programmers.

Such was my development in QBasic, foiled by lack of a way to share global variables between SUBROUTINE files. Why I didn’t just forget about SUBROUTINEs and move all the code into one file escapes me at the moment, but I suspect it had something to do with a stubborness or a lack of creative problem solving on that point. Or just not spending quite enough time with it. I was only 13 at time, with school and family taking precendce, so my attention to the problem was limited.

I did attempt another text-based game, this one a fantasy game with a main character called “Leonar” (becuase cool name), and did some other fiddling around with QBasic, but nothing else major resulted from that. My next set of adventures was with Evelop Basic, which is now more dead than QBasic.

Published April 28th, 2014

Jules

There are stories of rubber ducky (or teddy bear) problem solving, where the only thing that a person needs to do to solve a particular problem is articulate that problem to someone else, or in this case, something else. This leads to stories of engineers being able to solve their problems by explaining them to inanimate objects. When I explained this to my family, I suggested that I might want a rubby duck for Christmas, so that I would have something to explain my programming problems to.

So it was great fun to get a rubby duck for Christmas from my sister. She added a little story to it, projecting what Jules (the rubber duck) would be doing with me as I work on my programs. (She hadn’t read the wikipedia article that I linked to.)

Jules went through the papers, carefully picking up each sheet with his small orange bill. But Jules wasn’t really paying much attention to the papers. Nope. He was juggling complicated math problems and arranging and rearranging a line of ones and zeros in his head. He sighed; some problems are just too tough for a duck. This is what he’d been trained to do. But up ‘til know he’d only known the basics. And well, it was a lot easier to learn things in the training course.

Jules dropped the paper that was in his bill. He was tired. And the ones and zeros within his mind were in a dreadful brawl. A tall proud One had said to a pleasant chum of a Zero that he was fat! And well one thing had followed another…Jules yawned. He didn’t have the time or energy to sort it all out now. Jules set off clambering over mountains of paper. And twisting through mazes of knick-knacks that lay strewn across the desk.

Finally he came to a stop, looking at a crumpled snow white shirt. “Good,” he said quietly “it will do.” With that Jules started to wiggle his stout little body into the armhole. Jules now fully in the armhole had stopped wiggling and was resting his head against the soft material. It was warm and cozy. “Peaceful,” Jules thought as he slowly drifted off…

It’s awesome to have someone go the extra mile on a gift like that.

Published January 11th, 2014

Mechanical and Creative Work

I have a very active mind. As a home schooler, I read around 1000 books (of various lengths) between learning to read and graduating. I also tend to have at least one program/board game/story idea percolating in the back of my mind. I started programming so that I could build a few of those ideas.

The creative process carries a great deal of intrinsic motivation. But there is also a mechanical side to programming that can’t be escaped. The mechanical side of programming is where a program gets its polish, but it can feel a bit gritty and tiresome at points. I find that it happens when performing ports to a familiar technology, or when writing yet another simple* SQL Query (YASSQ).

This can cause problems when I need to do routine programming. Because the work isn’t intrinsically motivating by virtue of being interesting or challenging, I’ve struggled with giving it the attention and care that it deserves, especially when working from home, away from the motivation of colleagues.

I’ve often prayed and asked for prayer on this issue. Recently I saw a partial answer to that. I’ve discovered that I can do the least strenuous parts of programming while listening to podcasts. This struck me as similar to the way that routine math could be done with an radio drama in the background.

I don’t think this will work with all podcasts. Today I was listening to Ravi Zacharias’ various speeches and sermons while working. I don’t think I’d be able to listen to a technical or conversational podcast. Technical podcasts would make it hard for me to maintain a focus on my code, and I’m too much of a listener for the conversational podcast. Sermon podcasts seem to fit in a special spot there.

It’s also something that only applies to routine work. If I’m doing something that requires that creative side of the mind, the podcast has to be turned off (and is turned off promptly). Usually though, the work that I can’t do to a podcast is intrinsically more interesting. Of course, writing (like this post) is something that I can currently only do to either music or silence. Thankfully, I have grown to enjoy it a fair bit, hence this site.

*Or code that seems simple. Details are tricky that way.

Published August 19th, 2013

Performance gotchas in .NET: BindingList

Every programming framework has certain corner cases that can drain performance when mishandled. The .Net Framework is no exception. I’ve discovered a few in my work with C#. I’ll be blogging about these as I find them.

These are not micro-optimizations like using String.Concat instead of += "" over a small string. These problems will suck the performance straight out of your application, and are inconspicuous in the code, requiring a some kind of profiling to realize what’s going on.

One of these potential performance sinks is the BindingList collection, often used in Windows Forms projects when the form needs to be data-bound against a list of objects. It’s a handy class to have at your disposal. It gives the same methods for collection handing as the List collection, and binds to ComboBoxes and ListBoxes very easily.

There’s only one caveat: when BindingList is used incorrectly, it gets used in a Shelmiel the Painter algorithm. But it only does this under a certain set of conditions:

The first condition is that an empty BindingList (I’ll call it nameList) is bound to a ComboxBox or ListBox on your form.

var nameList = new BindingList();ComboBox1.DataSource = nameList;

The second condition is that the BindingList is then populated in a loop or by using AddRange():

names = get10000MaleNamesFromFile();foreach(string n in names)
{
    nameList.Add(n);
}
//OR
nameList.AddRange(names);

Put these two together, and you get really bad performance. For 1000 names, it takes around 0.2-0.1 seconds. For 5000 names, it takes a little over 2 seconds. For 15000 names, it takes my computer somewhere on the order of 20 seconds to add the list(see the download at the end to test on your machine). That gets bad if I every had to deal with more than about 5000 names, but it will also get worse if the collection in question has a lot of properties that are accessed, such that even 500 items could be too many.

What’s going on here is an unseen inner loop with the call to NameList.Add(). In order to make bindings work, the BindingList class exposes an event called ListChanged, which provides two parameters to methods that handle it: An object reference that points to the BindingList, and a ListChangedEventArg that gives some information about the list change. It seems that ComboBoxes and ListBoxes enumerate the entire BindingList every time that the ListChanged event is fired. (This is effect is difficult to log, I’ll update if I get a code sample that does so).

There are two approaches to fix this:

1) Don’t set the BindingList as the Data Source of a Combo- or ListBox until the BindingList is completely filled. This keeps the ComboBox and ListBox from enumerating over items multiple times and taking a long time.

2) Turn off the ListChanged events while adding a lot of items to the BindindList. Using the example above, this approach is coded like so:

names = get10000MaleNamesFromFile();//Turn off the ListChanged Events
NameList.RaiseListChangedEvents = false;
foreach(string n in names)
{
    NameList.Add(n);
}
//Turn them back on.
NameList.RaiseListChangedEvents = true
//Notify the changes to the ComboBox
NameList.ResetBindings();

Either of these approaches takes the amount of time into the millisecond range, where 15000 records takes less time than the list of 1000 names did before. Now the code runs with linear performance. For anyone that wants to experiment with this performance issue, I wrote a small Windows Form that demonstrates the issue: you can download it here.

Published July 27th, 2013

The People you Play with...

My brother and I rolled dice and commanded armies, testing our strategy against the wits and armies of Kaser the Kaiser, as he like to call himself. Wars waged went back and forth over the RISK board, to the sound of much hilarity, fun, and explanation of rules. My first RISK game, colored by nostalgia and entertained by the fun nature of Mr. Kaser, seems like one of the funnest ones I've had. The people you play with have a dramatic effect on your board gaming experience.

Since then I've played RISK several times, but over time I've realized that it is a game that is fundamentally drawn out. My relish for RISK has faded with experience. I've moved on to other games that have more intrinsic fun, like Dominion, Settlers of Catan and Race for the Galaxy. These games bring better pacing, balance, and variety to family game night. But you can't escape the importance of the chemisty with your fellow gamers.

Catan, for instance, has mixed results in a family setting. Catan's featured robbery and competition have to be taken impersonally for the game to stay fun. Some families work that way, others don't. Dominion is a bit worse in this regard, since it features multiple attacks that can be piled on the other players. Cooperative games like Pandemic and Forbidden Island help this a lot, but there are only a few of those that have been designed at this point.

These family concerns are part of why I enjoy attending the hardboarders gaming group. I can't speak for board gamers in general, but the Juggernaut group gives me a place to engage in board games for their own sake. I don't have to worry about hurting feelings by playing competitively. Everyone is here to relax and take on a small challenge or five. I get a chance to try new games and discuss games with like minded board gamers.

That said, there are some games best played with family or close friends. I find them to be the ones that give the most hilarity for effort invested. Telestrations is one of these. (If you haven't played it, or the pencil and paper version of it, you now have your next game night planned. I highly recommend it). It's basically telephone, but with paper and pencil. It'll bring out the strange side of people. And sometimes that the funnest side of all. :)

(Example thougts from a game turn of telestrations)

What is this drawing that I'm looking at? Is it Obi-Wan, or a woodchuck? And how on earth am I supposed to communicate hypocrisy in a sketch?

Fair warning: Sense of humor required, shared bizarre sense of humor optimal. Hence the strength of the game for families

Published July 20th, 2013

Well Factored Code : Calling SQL

Disclaimer: None of the SQL refers to data that I use at work. They were inspired by golang's database/sql package documentation

One of the things that I've been learning as a programmer is what well factored code looks like. There are several concerns that go into it, but one very important part of well factored code is the principle of Don't Repeat Yourself (DRY). The net result of DRY is that when you only have to write the unique elements of your code, removing repetitious noise. This helps you focus on actual differences in code, instead of trying to to filter the signal out of the noise.

This sounds fine in theory, but how does it look in practice? My day job involves a lot of SQL queries from C# code. There are several options, but one that I often use to is to use SQL in strings and Stored procedures calls via ADO.NET's SqlDataReader and SqlCommand classes. At a library level, these classes might be considered well factored, since all the variable parts of a SqlCommand are split into different classes:

SqlConnection, for connecting to databases
SqlParameter, for protecting from Sql Injection
SqlDataReader, for gathering the results of a SqlStatement
SqlCommmand, the glue that holds it all together(attach it to a connection, feed it parameters and execute it to run the command and/or get a SqlDataReader).

The problem with these classes is that if you are using each of them out every time you write a query, you end up with a lot of repetition. For example, below is how wrote my SQL calls in C# until recently (when I discovered the the rest of code in this article).

int age = 27;string eyeColor ="Brown";
SqlConnection conn = new SqlConnection("Data Source=myServerAddress;Initial Catalog=myDataBase;Integrated Security=True;");
SqlCommand getPeople = new SqlCommand("Select name FROM users WHERE age = @Age and eyeColor= @eyeColor");
getPeople.Connection = conn;
SqlParameter ageParam = new SqlParameter("@Age", age);
SqlParameter eyeColorParam = new SqlParameter("@eyeColor", eyeColor)
try
{
    conn.Open();
    getPeople.Parameters.Add(ageParam);
    getPeople.Parameters.Add(eyeColorParam);
    SqlDataReader results = getPeople.ExecuteReader();
    while (results.Read())
    {
        Console.WriteLine("{0} is {1}", (string)results["nane"], age);
    }
}
catch(SqlException ex)
{
    Console.WriteLine("Error: {0}", ex.Message);
    throw;
}
finally
{
    conn.Close();
}

(Yes, I know about using, but when you need to catch errors, finally works just as well)

There are a lot of issues with this code. First of all, if you forget any of the Sql* objects, you'll end with a runtime error. This has (for me at least) resulted in many testing sessions where I run code to find out that "A SqlCommand requires an open connection", or "Command expects parameter @Age, which was not supplied". It's also long, and this means that it will be copy/pasted all over a code base (naturally, nobody wants to type that code twice), and other logic will get mixed into it.

Clearly a helper method is in order. First we want to extract all the pieces that will usually do the same thing. For most applications, this ends up being the connection string. So we can extract that into a helper method. But we should go one step further. When fetching results from a SqlDataReader, you can call ExecuteReader with a CommandBehavior parameter, which allows you to tell the data reader to do different things with the command.

One of the options that it gives you is CommandBehavior.CloseConnection. This tells the SqlDataReader to close the connection when the reader is closed. This allows us to pass a SqlDataReader and not have to worry about the connection (as long as the reader is closed). These two pieces give use the following, which allows us to remove the code that manages the connection string.

public static class Queries{
    SqlConnection conn
    {
        get
        {
            return new SqlConnection("Data Source=myServerAddress;Initial Catalog=myDataBase;Integrated Security=True;");
        }
    }
    //Because the connection is passed in, we don't close it.
    public static SqlDataReader Run(SqlCommand comm)
    {
        comm.Connection = conn;
        //The caller needs to catch exceptions here.
        if (conn.ConnectionState == ConnectionState.Closed)
        {
            conn.Open();
        }
        return comm.ExecuteReader(CommandBehavior.CloseConnection);
    }
}

Also useful when dealing with SqlDataReaders is that you can iterate over them with foreach. This exposes a collection of IDataRecords that point at the reader. The reader is also automatically closed when the foreach loop exits, so we get automatic cleanup. With the CloseConnection behavior we don't have to worry about the connection either. Combine that with our helper method above and we get the following usage:

int age = 27;string eyeColor ="Brown";
SqlCommand getPeople = new SqlCommand("Select name FROM users WHERE age = @Age");
SqlParameter age = new SqlParameter("@Age", age);
SqlParameter eyeColorParam = new SqlParameter("@eyeColor", eyeColor)
getPeople.Parameters.Add(age);
try
{   

    foreach (IDataRecord r in Queries.Run(getPeople))
    {
        Console.WriteLine("{0} is {1}", (string)results["name"], age);
    }
}
catch(SqlException ex)
{
    Console.WriteLine("Error: {0}", ex.Message);
    throw;
}

That clears up several of the Sql- objects that we were dealing with before. We also lose the finally block, as Queries.Run takes care of the connection. But there is potential for more factoring. Why do we need to have the SqlCommand getPeople if we are just going to add a parameter to it and build it from a string?

This suggests a slight modification to the Queries.Run function above. Instead of taking a SqlCommand, it can take a string and an array of SqlParameters, similar to the way String.Format works. I used to params keyword to save the caller from always having to explicitly build an array from the SqlParameters.

public static class Queries{
    SqlConnection newConn()
    {
            return new SqlConnection("Data Source=myServerAddress;Initial Catalog=myDataBase;Integrated Security=True;");
    }
    //The caller is responsible for catching exceptions
    public static SqlDataReader Run(string query, params SqlParameter[] parameters)
    {
        SqlCommand command = new SqlCommand(query);
        command.Connection = newConn();
        if (parameters != null)
        {
            command.Parameters.AddRange(parameters);
        }
        //The caller needs to catch exceptions here.
        return command.ExecuteReader(CommandBehavior.CloseConnection);
    }
}

This gets used like so:

SqlParameter age = new SqlParameter("@Age", 27);SqlParameters eyeColor = new SqlParameter("@eyeColor", "brown")
try
{
    string q = "Select Name from Users where age = @Age"
    foreach(IDatarecord user in Queries.Run(q, age, eyeColor))
    {
        Console.WriteLine("{0} is {1}", (string)user["Name"], age);
    }
}
catch(SqlException ex)
{
    Console.WriteLine("Error: {0}", ex.Message);
    throw;
}

Now we only have 6 lines of code* for the same result as the 16 lines we had before. And we have a method that we can use anywhere we need to pull some results from the database a bit of C# connects to. I think this is a decent factoring of the process of calling SQL code from C#, but it is a bit incomplete compared to ADO.Net. It doesn't handle the need to just run a query without fetching results, or running Stored procedures directly. It could also use a way to run a command with a specific timeout.

It's important to note that adding those to the mix would change how the code should be factored. Since our simple example is just pulling brown-eyed 27 year olds, we don't need that extra stuff. And that's one thing to realize when factoring code: only factor as much as is useful. Beyond that, and you have extraneous yak shaving.

Published July 11th, 2013 Edited on September 30th, 2019, to switch from using positional data reading to name-based data reading.

Skill Is Not Calling

In the Kingdom of God, obedience is worth more than skill. God is more than able to make up for our deficiencies. These are some things that I've known to be true on a mind level, but lately God has worked in me to help sink into my heart. When we go on our own skill, the best we can do is be something that the world could have produced on it's own. Thus we have people that do "good" things, whether by temperament, a guilty conscience or civic duty. We end up reflecting ourselves, in all our human fallibility, and will disappoint those who see us.

But God has not called us to run on our own skill. He has called us to obey his calling. His calling is not always comfortable, nor is it always tame. Regardless of how strange or difficult his calling is, he will provide for it to be fulfilled. When we are in tune with the Holy Spirit and obeying him, we don't run on the stuff of this world. The stuff of this world doesn't produce superhuman results.

Instead we run on the power that comes from daily relying on God. His daily grace empowers us to do things that aren't humanly possible. That is where we will see the world changed. It is this grace that allows the Brooklyn Tabernacle to reach out to the brokenness around them. This grace that holds Christians up under persecution. This is the grace that calls missionaries to obscure countries to translate the Bible for languages without written form.

And when people follow this calling, it turns heads. Reactions vary from amazement to befuddlement to anger. As a missionary and preacher's kid I've witnessed the power of people following God's call. To see the heart and anointing of a person surrendered to Christ is an amazing thing. It leaves a hunger for more than man can do on his own. For those who accept it, it shines a hope for greater things in this world of darkness. And that is a hope that I am grateful to have, and want to share.

Published July 1st, 2013

About Me

Who am I?

I am a Benevolent Outsider. I grew up everywhere and nowhere. I am a young fish who can see the water of America. I have been Other, in many places I don’t belong. I welcome people with dignity, because they are human. I give the benefit of the doubt, and wish the best I can for those I encounter.

I am a Well of Tales. My life is a long story, and I gather the threads in the lives of others. My mind weaves them into a beautiful, terrible tapestry, covered with struggle, joy, secrets kept in confidence, lessons learned, tales of wonder, vistas, and beautiful souls.

I am a Caregiver. Those I love, I love dearly. I tend to their needs, gathering resources, providing a shoulder to cry on, a validation of their traumas, a safe place to vent. I tend to care tasks, however ineptly, because I do care. Because those whom I love matter to me.

I am a Translator. I seek to give a voice to those in a foreign land, to communicate with and for those who struggle to find words. I understand that language is nuanced, that context is important, that life is rarely as simple.

I am a Student. I am never finished learning, few of my opinions are final. I do my best engage with ideas earnestly. Knowledge is always subject to revision upon new evidence. I present my knowledge with an appropriate level of confidence, and source my learning, that others may draw deeply from the wellsprings of knowledge I find.

I am a Creative Voice. I express myself in code, work, and pixels. I have an emerging, if, incomplete style in these media, a desire for clarity and whimsy. I embrace silliness, because life is hard enough without also burdening myself with a heighted self-seriousness. I follow the muse into the night.

I am a Poet. I avail myself of the magics of rhythm and evocation. I believe that words written, spoken and read have power, that getting them right matters. That is it worth seeking the quality without a name that elevates ideas and stories above mundanity.

I am yumaikas, raised in the ocean, jungle, mountains, and heat.