tag:blogger.com,1999:blog-371204262024-03-08T08:52:18.462-08:00Jim Kingdon on ProgrammingJim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.comBlogger57125tag:blogger.com,1999:blog-37120426.post-1770034428244046972023-04-21T13:21:00.000-07:002023-04-21T13:21:42.884-07:00Non-functional requirements<p>What about security? What about accessibility? What about visual design? What about usability? What about data analytics? Do we have to write tests?<br /><br />A very effective way for a project to die before it even gets started is to load it down with a lot of "well, you can't start until you address X" for dozens of different topics X. Yet we've also seen the dangers of worrying about everything later. After a big system has been built it can be hard to try to retrofit (especially if we do not communicate what is not present or carelessly use words like "done"). Can you just ignore the annoying topics unless you are sure they are biting you? Up to a point yes, but I'm operating from the assumption that you want to protect our users data, you don't want your users to be baffled, and the like.</p><p>Are these core or extras? There's a lot of judgement which goes into this. It will depend on the specific topic and we need a way to make those decisions and some idea of where that fits into the whole process of getting our software into the hands of users.<br /></p><p>When do you address non-functional requirements? I'd say as you build the software. Not only is this more manageable than an up-front-focused process, it is also more effective. Making a lot of plans about how secure (or accessible, or operational) your software will be is only as good as your follow-through, so develop and revise your plans or techniques as you are implementing. Not only will this be more feasible, the presence of running code will give a reality check and a degree of concreteness which will improve your ability to find the best ways to achieve your non-functional goals. Make it a habit, not a totally separate process.<br /></p><p>One good technique for some topics is to check for desired behaviors in your testsuite which gets run regularly (typically on every pull request or commit). Open source linters exist for topics like accessibility and security, and in many cases you can write your own (they don't need to be perfect to be useful, as a simple text search may be good enough for something like whether you are, for example, calling the method which sends logging to your centralized logging service rather than the logging method which does not). Bring the people along too, because it is no fun to keep automated checks passing but give no thought to whether those checks achieve their intended purpose. But the automated checks are fairly easy to implement and conducive to a situation where software is changing constantly.<br /></p><p>It can be daunting especially if you have a small team. "I don't know anything about security!" "I'm just a graphic designer, not a UX expert!" "Why can't we just write our logs to disk?" "Who are we publishing these metrics for anyway?" Don't let this paralyze you but do try to build awareness within your team as you can and also get help as you are able. For example, in the area of UX, having empathy for the users and just asking the question of what they are trying to accomplish will be a good start. More broadly, think about how you'll know whether you are doing a good job (for example, the role of penetration tests in the security landscape is a whole topic of its own - but the idea of a penetration test originated from a good impulse, of trying to find out how secure your software is rather than just operate on unverified assumptions).<br /></p><p>Also, give some thought to your definition of done. In many contexts your compliance department has some rigid-sounding rules about characteristics your software must meet before some point (maybe before any user uses it, maybe before it is generally available, something like that). Try to make yourself rules which fit with those or exceed them. And apply them at the same level as you do for other requirements (often before considering each user story done).<br /></p>Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-38806902946107119392023-04-18T22:18:00.001-07:002023-04-18T22:18:39.512-07:00If you support it, you get to enhance/replace it<p>Are you a software creator or a software maintainer? If this sounds like a trick question, you might be on what I'm calling a build/operate team. I'm actually not sure whether there is a standard term for this. It is at least similar to a "product team" as opposed to a "project team".</p><p>So we are talking about a team which owns a particular sort of value and is staffed to provide it, including as much of product, design, engineering, testing, support, etc, as feasible.</p><p>One way to say this is "if you build it, you support it". In that case, a handoff from a build team to a maintenance team is an anti pattern.</p><p>But when I showed an early draft of this essay to someone struggling with these patterns, they objected. But we don't want everything to be owned by the last person/group that touched it! We don't want to make it impossible for someone to chip in without committing themself unto generations to come!</p><p>That's what made me think of flipping "if you build it, you support it" on its head. What if we formulate it as "if you support it, you get to enhance/replace it"? Especially if your organization always seems to neglect maintenance activities, putting them at the front of your mindset may be helpful, but this flip also helps address some of our paradoxes from before.</p><p>Is something owned by the last person who touched it? Not really, we're aiming for a world in which someone who jumps in temporarily is working with an owner who is engaged enough to understand what is being done, has the final say on how it is done, and knows how it ties into their ongoing responsibilities.</p><p>Can we reorg without damaging the principle that people build things and also support them? Yes, although to follow the rules a reorg has to assign the ongoing tasks which come up on a regular basis, as well as the glamorous new things which we are getting all excited about.<br /></p>See also:<p>"Products Over Projects", by Sriram Narayan, https://martinfowler.com/articles/products-over-projects.html .<br /></p>Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-33471015607813298012023-03-17T14:44:00.000-07:002023-03-17T14:44:31.499-07:00Technical design: whether, who, how, and what <p><i>This is a lightly edited update to this post originally published on 20 Aug 2018
</i><br />
</p><h2>Do I need a technical design?</h2>
<br />
In agile software development, there is architecture (decisions that are
hard to change) and incremental design. Architecture, in this sense, is
a pretty small number of things—programming language and probably
application frameworks and data storage. Incremental design is the norm:
we add classes, endpoints, and database tables as we identify a need
for them, or remove them as they are unneeded or replaced.<br />
<br />
But what about decisions in between these two extremes? For example, it
used to be that users all signed up for the website as individuals, and
now there is a need for some kind of organization which can manage the
users under it. Or we used to have a bunch of separate products with
their own logins, apps, and management and now there is a need to do
some or all of those things in ways which apply to all products. Or our
application used to assume that all users needed to be connected to the
internet at all times, and now we want to build in offline operation.<br />
<br />
I won’t completely rule out handling larger changes via the usual
communication of incremental development—pair programming, discussion of
individual stories, pull request review, and the like. But it can be
hard to maintain a clear idea of the larger design that way, and I have
usually been happier with a discussion which happens at a higher level
and whose goal is to get a direction into which we can fit in the
smaller decisions that we will make as we go.<br />
<br />
I’ll write more later about who should drive this process, how to
develop such a design, and what is worth writing down and communicating.
But let's first ask when we should be
doing this design.<br />
<br />
It is tempting to say that the high level design of a system must happen
before we can start breaking down the work or implementing pieces of
it. Which sounds good, and is nice when it works out, but I have yet to
see a design of this sort which does not get changed during
implementation. There’s a lot of reality check (interactions with
existing functionality, feedback which we only get when we have an early
version to show, complications which we didn’t notice at first).
Therefore I wouldn’t try to finalize the design before we start acting
on it. And I wouldn’t go to the other extreme—of trying to make major
changes in a fully incremental way and doing all the communication after
the fact. My preference is to start with rough ideas and conversations
about the design, and as those get refined and conversations continue,
there is a point where the general contours start falling into place.
That’s about when I start implementation. I want at least some of the
coding to be happening (even if we know we might be revising it later),
because otherwise I don’t really trust the design. In parallel, I’m
stepping up the communication (documents, meetings, etc). As things fall
into place (which may include allocating people’s time, agreeing on
technical or business decisions, and getting a clearer picture of
implementation choices), you’ll fall into the rhythm of building the
thing, because the general contours of what you are building have been
established by this point.
<h2>Who drives a technical design?</h2>
<br />
So we have a problem which is large enough that we don’t think we want
to approach it in a purely tactical way, and we’ll even assume we have
defined at least the general outlines of what we want this design to
accomplish. Who should turn this into a design detailed enough to
implement?<br />
<br />
Before I discuss who, let me say this is an intrinsically messy process.
There are a bunch of things we want out of our design. Things to do now
or save for another day. People (in various roles) with opinions
(either because, well, people have opinions, or more nobly, because they
have a specific organizational goal they are trying to achieve). See
for example Gregor Hohpe’s <a href="https://martinfowler.com/articles/architect-elevator.html" style="text-decoration: none;">The Architect Elevator</a>.
Issues like reliability, security, accessibility, and branding. A large
design space (a distinguishing character of software being its
malleability—or at least potential for malleability). Pros and cons for
pretty much every aspect.<br />
<br />
If that seems daunting, don’t despair. Just don’t be surprised if a
decision which was discussed at length, carefully considered, agreed by
all, and signed off subsequently starts to seem less settled. Or someone
who you had thought was aware of what was going on suddenly “discovers”
your design and has suggestions. Or your scope seems to keep expanding
or contracting.<br />
<br />
The most important person in this process is the one who is refining the
design and who will be involved in implementing it. We can call them
the “<b>responsible</b>” person (although don’t think of the roles too
rigidly—I did say this process tends to be on the messy side, didn’t
I?). To do all these things, and have time for this design, the
responsible person needs to be able to focus on this (usually, this
means they aren’t a manager).<br />
<br />
But that person can’t produce a good design by sitting in a room and
thinking hard (if for no other reason, because getting buy-in is a key
part of what will make this design get implemented and achieve its
goals). Therefore their main activity is going to be communication. I’ll
talk later about how to communicate and what to communicate,
but in the context of “who”, identify who should be “<b>consulted</b>”.
That is, who needs to be aware of the design and would have good ideas
about how to do it. Broadcasting what you are doing and inviting input
works well, but I’d also directly seek out the people who will be most
knowledgeable or important.<br />
<br />
One rule of thumb for involving a lot of people is “accept input widely,
accept direction narrowly”. You want to hear from as many perspectives
as you can. Whether or not you take the advice, thank people and
appreciate that they took the time to engage with you. These will be the
people who help communicate the changes you are making.<br />
<br />
Saying “accept direction narrowly” raises the question of who ultimately will be deciding. This role is generally called the “<b>approver</b>”
and will often be the manager of the responsible person (the details
will depend on your organization, though). Sign-offs are a good way of
formalizing decisions already made and making sure that there is
sufficient buy-in throughout the organization. They aren’t good at
exploring different possible solutions or weighing pros and cons, so
think of formal sign-off type processes (if you have them) as a way of
ratifying what is already understood, not as a way of hashing out
agreements.<br />
<br />
Lastly we have people who aren’t necessarily providing input but who should be “<b>informed</b>”
about the design. The basic goal here is to cast as wide a net as
feasible (in accordance with “err on the side of overcommunicating”
which tends to be good advice especially in larger organizations). Think
of ways to reach a variety of audiences: different levels of detail,
different ways of presenting the work (for example, it can work to have
one document which is technical and one which is more about the business
goals and rationales—as long as they are reasonably in sync on topics
such as what is in or out of scope), or different places you can
announce what you are doing and offer to answer questions or sync up
with interested parties.<br />
<br />
Describing the responsible, approver, consulted, and informed roles
makes it clear that communication is central to the process of making
technical decisions and being ready to put them into practice. The next
two parts of this series will be about how to communicate, and what
topics to include in that communication.
<h2>How do I develop and promote my technical design?</h2>
In the first two parts of this series we figured out we needed some kind
of technical design, and we figured out who should be making that
happen. How does the responsible party get this thing going? Do you call
a meeting? Write something up?<br />
<br />
Typing “useless meeting” into an internet search engine and reading the
results should be enough to give us pause about calling a meeting to
hash out our technical design. Yet in so many organizations the meeting
is the mechanism by which attention is allocated, or is otherwise
necessary. So first, what are the pitfalls? The usual risk of a meeting
turning into (too much of) an open ended discussion is exacerbated by
the large design space and many stakeholders. Another sign that meeting
discussion is a bad idea is if the wrong people are there: don’t
hesitate to say “can the three of us (less than the whole meeting) have a
break-out on this topic after the meeting?” or “would you be willing to
talk to X (who is not present) and bring the information back?” Set
your goals, such as (1) make a brief announcement about what is underway
and how people can get more details or engage further, (2) present your
design to date and solicit clarifying questions, or (3) give people an
opportunity to raise concerns to be addressed in the future. Or if you
do want a longer discussion, set the topic, keep an eye on the clock,
and don’t be afraid to steer the group back to the agenda. Also, aim for
a level of detail appropriate for the people in the meeting. Software
developers may be most interested in database schemas and code
organization, infrastructure engineers may be most interested in
reliability, security or how your design is spread across various
machines, product may be most interested in what functionality your
design will or will not unlock, and so on.<br />
<br />
I’ve often had good luck circulating the design in document form. People
have something to react to and can leave comments on the document
itself or in other ways. So is this a Big Design Up Front? Not exactly.
I’m aiming for something closer to a High Level Design Written As We
Need It. It is at a higher level than code. It is at a higher level than
detailed descriptions of functionality (click on button X and see the
following fields with the following error conditions). It might contain
things like database schemas or protocol specifications, although
sometimes even that can be a bit fine grained.<br />
<br />
What is a design document for? First of all, as a communication tool.
Secondly, to clarify the thinking of the person writing it. What about
things like traceability between requirements and implementation,
justifying the need for making a change, or documenting what has been
changed? I would tend to think of those kinds of documents (how many you
need will vary depending on your situation) as separate. The design doc
is written and revised as you are thinking something through and
figuring it out. More concrete documents (including breakout into tasks,
specifying behaviors in detail, or explaining code details), have a
greater need for detail and precision and are the output of the design
process, although of course the design document can link to them as they
are created. Seeing the design document as a communication tool helps
focus the process of writing it. Imagine that it is a conference talk
and you are trying to figure out who is the audience and what they would
want to know about your design.<br />
<br />
Expect to iterate on the design. Gather some ideas. Think about them and
boil them down to a proposed design. Talk to people one on one.
Circulate it in writing. Figure out how else to get it out there. That
will generate ideas and reactions. Figure out what to revise based on
that. Expect to repeat this process until there is a sufficient degree
of convergence on a course of action. Don’t fall into either the extreme
of spending all your time talking to people (and not getting around to
taking in what they said, researching things as needed, and making some
decisions), or the other extreme, of thinking through something and
coming up with something which makes sense to you, but which may lack
buy-in from other people or may miss important requirements.<br />
<br />
So we are developing our design and communicating in diverse ways
(presentations, written documents, informal discussions, and yes maybe
even meetings). But what topics should we cover? The last section goes into some specifics.
<h2>What goes into a technical design?</h2>
So far we decided we need a technical
design, figured out who would be doing it, and how we’ll be sending it
out and getting input. But what is the content of that communication
(for example, what sections would we put into a written design
document)?<br />
<br />
What to include will vary depending on your organization and the needs
of a particular design. For an early stage startup, anything relating to
scaling and operations may take a back seat to “am I building something
people want and how can I most quickly validate my hypothesis?”. For a
company in a highly regulated space, there may be a lot of requirements
specific to your field.<br />
<br />
The same applies to an individual design. Does my design concern a
server with a high or low need to be available? Does my design concern
data which is sensitive? Does this design change anything related to
this topic? (If not there’s probably little to say on the subject). For
that reason, I’d suggest treating templates (including this article) as
guidelines, and omitting sections which don’t seem relevant. One of the
fastest ways to lose an audience is to include a bunch of material that
you aren’t very interested in (and probably didn’t do a very good job
with). And of course to prioritize everything is to prioritize nothing, a
good motto in a variety of contexts.<br />
<br />
So, what might we include?<br />
<br />
<div>
<b>Goals and non-goals</b></div>
<div>
These are perhaps the most important sections. If you can figure out
what your design achieves and what you are leaving for another day or
deciding is not worth doing, you are well down the path of figuring out
how to do it.</div>
<div>
<b>Description of the proposed solution</b></div>
<div>
What changes will we make to code, data, networks, and hardware? How
does this design achieve the goals? Give enough detail that people can
see some of the implications of various choices, but try to avoid the
kinds of details which can easily be fleshed out during implementation.</div>
<div>
<b>Security</b></div>
<div>
What data is stored and sent where? How is access controlled? If
cryptography is involved, how are keys managed and have we chosen
appropriate algorithms? Are some parts of the system isolated from
others and if so how?</div>
<div>
<b>Reliability</b></div>
<div>
Is there redundancy? What are the consequences of network outages? If
data is stored in a primary-replica setup, how do we choose a new primary? If
data is written multiple places how do we reconcile them? Are there
rate limits or other ways of keeping a problem one place from cascading
elsewhere?</div>
<div>
<b>Capacity</b></div>
<div>
What is the expected load on the various systems involved? Does load
ramp up gradually or do we expect a sudden spike in traffic? What needs
to be handled manually and is there sufficient staffing to do it?</div>
<div>
<b>Monitoring</b></div>
<div>
Do we need to report new metrics? How will we know about errors?</div>
<div>
<b>Data analytics</b></div>
<div>
How will we measure usage of the new functionality? What kind of analysis might we want to do?</div>
<div>
<b>History</b></div>
<div>
Has the company considered this problem before? What previous decisions
got us here? If there are documents describing previous designs, I tend
to just link to them rather going into a lot of detail about what has
gone before.</div>
<div>
<b>Storage</b></div>
<div>
What database(s) are involved (new or existing)? What changes in database schemas are required?</div>
<div>
<b>Interfaces between systems</b></div>
<div>
Defining these can help clarify the design and is particularly helpful
if one of the functions of your design is to coordinate between
different teams or companies who are responsible for different pieces.</div>
<div>
<b>Alternatives</b></div>
<div>
How else did we consider solving the problem? Why did we choose the solution we are proposing?</div>
<div>
<b>Open questions</b></div>
<div>
This section is particularly helpful if you know certain topics are
controversial or warrant further discussion. As questions are resolved,
move items from here into the main design section or the alternatives
section.</div>
<div>
<b>Rollout</b></div>
<div>
In what order are we building this? Are we shipping it continuously? In a
series of phases? Is it rolled out selectively to certain users?</div>
<br />
These questions can be taken as a template for a design document, but
they also can be used to figure out who to go talk to, what to put into a
presentation, or what anticipated questions to prepare for.<br />
<br />I've talked a lot about things to do: Did you
talk to X? Did you consider Y? What if we did Z? And those are all very
helpful up to a point. But only do those things which seem necessary for
your particular organizational culture and problem you are trying to
solve. The purpose of all these suggestions is to help you build things
and solve problems, so as you go, don’t be afraid to keep asking
yourself and others: Are people on the same page now? Is this enough
specificity to build this? Is my technical design sufficient for what I
need?
Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-23695209736537710232021-06-28T11:59:00.000-07:002021-06-28T11:59:04.187-07:00My company got acquired! Now what?<p>So you've been working on this product, got something interesting, probably even some customers, and a larger company got interested in the product enough to acquire the company (some of this advice also applies if they were interested in the people or something else other than the existing product, but I'm mostly writing here about the case where - at least supposedly - the intention is to keep and grow the product you were working on before the acquisition).</p><p>Some of the following is broken down by function but even more than usual this is a case where it pays to have some cross functional awareness of what is happening even if you are responsible for one of these areas more than others.</p><p>First of all, you are now part of a much bigger company. Therefore, a
lot of the challenges are communication ones. There's a whole set of
issues around getting to know people in disparate parts of the
organization, setting expectations, self promotion, and probably a bunch
I'm forgetting to call out specifically. But many of them change and
get more important at a bigger company.</p><p>An acquisition tends to bring up a lot of feelings - for example excitement, accomplishment, sadness, and disorientation. Particularly for people managers, but also everyone, a lot of the job is talking people - including yourself - down
from various ledges and getting information about benefits,
offices (and/or remote practices), org charts, company strategy, and more. Basically to make sure people know what is
going on, have input as feasible, and that things like 1:1s are doing
the job of getting into issues which might be less amenable to blanket
emails and the like.</p><p>For product, the key challenge is "are we working on what the higher ups acquired
this product for?" But before even getting to that, regularly ask a more basic question: "Do people widely understand our product (existing and
future functionality)?" The big company adage is "err on the side of overcommunicating" and my experience is that you need a lot of different ways to even get to a shared baseline of what we have today (for example, via demo days, screenshots, getting people internally to try the product, and bringing in customer input). Many of the same mechanisms also apply to a shared understanding of what we want to build next and why.</p><p>For technical issues, how does the other company handle deployment? Security?
Programming languages? Testing? How much do we expect to standardize
and how much do we expect to remain divergent? If we want to converge,
what do we tackle first and how?</p><p>Culturally, the number one thing I'd focus on is how to have contact with people who had been from the other company. Maybe there are interest groups around hobbies, diversity, or charitable activities. Or more work-related things like "people using a common programming language", "people interested in security", or other concerns which may cut across the org chart. It can be easy to neglect things which aren't tied to a concrete deliverable, but the goal here is to build relationships. It is so much easier to navigate an unfamiliar organization and solve a tough problem if you know people and understand assumptions or typical ways of approaching things.</p><p>Will you stay with a company for long after acquisition? Does a product have a good change of thriving after acquisition? I'm not diving deeply into that, and there is no shame if the company you end up leaving in a few (months, years, whatever) just doesn't feel like the one you worked for pre-acquisition. But here I try to present the optimistic case for how you can jump into tackling a work situation which just changed (perhaps very dramatically) when your company was acquired.<br /></p>Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-26946695921109141532021-05-09T09:19:00.003-07:002021-05-09T11:59:22.896-07:00The Mortifying Ordeal of Soloing All Day<p>I read <a href="https://www.simplermachines.com/the-mortifying-ordeal-of-pairing-all-day/">The Mortifying Ordeal of Pairing All Day</a> by Nat Bennett, and.... well first of all it is a good read and worth trying to take in. Perhaps the best way to respond with such a heartfelt and personal story is with my own story. Maybe some day I'll find another way to tell it, but it rang surprisingly true to just take that article and instead lightly edit it to be about my own struggles with soloing (including in many companies where pairing had once been a norm but then fell away for various reasons). The result takes a few liberties here and there but is on the whole autobiographical.<br /></p><h3 style="text-align: left;">The Mortifying Ordeal of Soloing All Day</h3><h4 style="text-align: left;">I had to confront a lot of my fears about myself, sometimes every day. I
had to learn to show someone else all the things I didn’t know, my
limitations as a human and a software engineer.</h4><p style="text-align: left;">From 2014 to 2020 I was part of an experiment: I soloed all day, most days, for years.
Hundreds of other engineers joined me in this experiment. I was working
as a software engineer for some of Tech's most exciting startups, and everyone soloed, often for eight hours a day.</p><p>This was one of the best things I’ve ever done for myself, socially
and emotionally, and it produced some great software. It also burned me
out. Not “I don’t want to think about work” burnout. Probably not even “I don’t
want to work ever again” burnout. Whether "burnout" is even the exact best word is unclear, but it is close enough to describe a situation where I was often worrying about work (not sleeping well for example).</p><p>I spent much of 2020 in discussions with management about how I was doing which led to leaving my job in May 2021 with no plan much more concrete than "give myself and the world a few months to breathe". I'm still recovering to the point of being able to set goals for a job search (or other plan for the future).<br /></p><p>I also believe that the expectation that everyone solo, all the time, led to technical and product failures at multiple companies.</p><p>There’s a response I often get at this point, especially from people who were managers in the organization at the time:</p><p>“But Jim, teams weren’t expected to solo all the time. You might have been assigned tasks, but you were given leeway to accomplish those how you wanted, and management didn’t <i>require</i> people to solo all the time. If a team wanted to solo less, they could.”</p><p>This
is true. Engineers and teams had a lot more freedom than they realized they
had. I spent a lot of my time there helping people realize that. I would
often spend one or two days a week pairing for at least part of the day.</p><p>And yet.</p><p>Engineers had individual laptops and sometimes set up them according to individual preferences.</p><p>We were Tech Company Engineers, and one of the things that made us Tech Company Engineers was that we soloed.</p><p>One
of the great and terrible things about Tech, is that it operationalizes
peer pressure. It harnesses drives that humans have, drives for identity
and belonging, in the service of producing software. These forces were
only tenuously under management control.</p><p>So I soloed all day, most days, for about five years. This had a <i>lot</i> of upsides, far more than I can list here. Soloing really develops being able to plan out and execute a task, giving yourself time to think through a problem, understanding the technologies you are using, and developing proficiency which might not happen if you are leaning on your pair (sometimes more than you realize). The impact of soloing, especially soloing that much, goes much deeper than its impact on the code, on the
particular work the team delivers that week.</p><p>(Here would go an anecdote about how people who solo have self confidence and mastery.... sorry I'm not thinking of an immediate analogue to the Overcooked example in the original article).</p><p>We take the time to understand a problem so we can make informed decisions.</p><p>This is the
real power of soloing, intense soloing. Understanding what you’re doing,
and adjusting it based on further research or the results of experiments, becomes automatic.
For someone who thrived on pairing and loved it when there was a strong team spirit, this
was an almost psychedelic experience. I transcended the limitations of the people I was working with and discovered that I was able to accomplish things.</p><p>I
remember once, looking over a large office filled with people working at workstations, and thinking, “This is the most talented collection of people I have ever seen in one space.” I understand why engineers so fiercely defend their right to hack.</p><p>But: cognitive impairment.</p><p>Months where I struggled to meet my own basic needs.</p><p>Soloing requires putting up a facade of self-sufficiency, to management and the rest of the organization, for hours
at a time. Being able to manage oneself, both physically and mentally. I had to manage my space, my
decisions, my thought processes, and often my feelings on my own.</p><p>This
never stopped being draining. Even with an easy team, where I had clear goals and could
accomplish things without effort, soloing well
requires staying engaged with my environment, with what I'm supposed to be doing. No
retreating into sensory experiences, no checking my phone, no wandering
off or getting distracted. Maintaining that level of focus for hours at
at time was thrilling, but it also required a serious exercise of will.</p><p>There were some teams where it required more than will. I had to
fight to stay engaged. I had to develop skills. There were people with
whom I disagreed, but with whom I struggled to resolve those agreements.
There were people who didn’t put nearly as much thought into my
experience as I was putting into theirs. There were people who expected
me to “just make it work”, despite an ill-defined and not-yet implemented interface that I was expected to use. There were people
who just made me anxious or uncomfortable.</p><p>I had to confront a lot
of my fears about myself. I had to learn to show someone else what I could and could not do, my limitations as a human and a software engineer.</p><p>Over
time, over years, soloing wore me down. Took a little bit more each day
than I could recover. Until my life was working, and recovering from
work, and then working some more.</p><p>And then the pandemic happened. Overnight, suddenly, I was performing
this daily act of will without the support of the office, without going out to lunch at my favorite restaurant, without anyone to talk to except in scheduled meetings,
while the world burned down around me.</p><p>I crumpled. I stopped being able to solo. Stopped being able to have a conversation.</p><p>This
wasn’t everyone’s experience with soloing. In a two-by-two grid where
one axis is “sensitive to the demands of soloing” and the other is “time
committed to soloing” I’m hugging the upper left hand corner. My
experience was extreme.</p><p>And yet.</p><p>Tech companies, I’m told, have reputations as “burnout factories.” Many
people left my companies, at least a few of them to escape from the demands of daily soloing. People who love soloing, who see the benefit of it, but who
despite all those benefits are <i>tired.</i></p><p>There are people
who can solo indefinitely, for years. Who don’t experience the most
demanding version of it often. Whose recovery capacity comfortably
outpaces the demand. A lot of those people, at most tech companies, end
up in management roles, in leadership roles, and then they miss coding. Many places I've worked have had a leadership staff that, even when they believed me
when about how demanding soloing was for me, couldn’t really see it
themselves. Some of them even tried to find ways to enable me to solo less, but I'm not sure they understood all the forces which were making it hard to do anything but solo.<br /></p><p>We’ve all heard the bad reasons not to solo. "You are just cowboy coding." "You can't think about anyone other than yourself and your pet project." "You don't care about doing things well."</p><p>I’ve dismissed people making those arguments as fundamentally dogmatic, unwilling to do the hard work of real software development.</p><p>Now, though, I hear those objections, and I hear fear. A fear that I
share. A fear of exposing my vulnerability, my ignorance, my soft parts,
and a fear of the cost of that exposure, of the cost to my mind and my
body of subjecting myself to that exposure, day after day, in exchange
for a paycheck.</p><p>Underneath the urge to dismiss those concerns, I
hear another fear. A fear that soloing is too hard, that people wouldn’t
choose to do it if they weren’t corralled into it by individual goals and
hiring and promotion processes which emphasize "yes, but what did you do personally?". That a “soloing culture” is such a delicate wisp
of a thing that if you allowed engineers to solve problems together, they
would abandon soloing immediately.</p><p>What did we miss out on, by failing
to make more space for people not to solo? By treating this soloing
culture as something so fragile, and so precious?</p>Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-26259543992202663132021-04-23T20:09:00.000-07:002021-04-23T20:09:34.808-07:00Flow<p>So if you've been around tech, you've probably heard of flow. You know, the state of concentration where you are one with the code, getting stuff done, and the code loves you back. This is also often said to be incredibly fragile: the slightest interruption will cost you 15 minutes, in terms of how long it takes to resume your concentration.<br /></p><p>Although I've experienced this enough that I think I understand the process described here, my own experience of flow has much more to do with other people. I use the analogy of improv comedy - I suggest a thing, you say "oh, but what about this other component?", I say "well, let's try an experiment to confirm that idea", you say "ah, and that means we can delete this part of the code" and I say "oh and now that's gone, we can implement this other thing in a more elegant way". If you've heard of the "Yes, And" concept (which originated in improv and has been borrowed to workplaces) this may sound familiar. We're riffing off each other, we're generating ideas and figuring out where to take them, we're bringing in multiple perspectives about what we are trying to accomplish and how we are doing it.<br /></p><p>I mentioned this analogy to a co-worker who is a musician in their spare time. And they immediately said "oh yeah, jamming with someone is definitely a thing" (and contrasted it with teaching, which can feel quite different).<br /></p>Everything so far is based on my own experiences and folk lore within software companies. To write this post, I figured I should at least look a bit at academic or popular writing on flow, which seems to start with psychologist Mihaly Csikszentmihalyi who originated the term in 1975, and proceed to follow-on work in the following decades. Well, I've only read a few short summaries, but what I did surprised me a bit. It didn't neatly fit into the model of flow being all about being solitary and avoiding all interruptions. As far as I can tell it has just as much to do with whether goals are clear, whether the skills needed are within reach (with perhaps a slight stretch), and whether there is immediate feedback. And that flow can be either individual or group. I'd already decided that I didn't need to be threatened by the concept of flow even if it seemed to be promoting a working style which tends not to work well for me. That is, that I could redirect the concept to something which is recognizable but closer to what makes me thrive. It was nice to see that what is written on flow turns out not to be quite as different from my own thinking as I had initially imagined.<br />Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-29167244873611331362021-04-07T21:09:00.000-07:002021-04-07T21:09:28.196-07:00The four kinds of developers<p>I'm probably going to go to hell (or worse yet, business school) for presenting it this way, but developers (or probably more accurately development tasks) fall along two axes:</p>
<p>One axis is gregarious/solitary</p>
<p>One axis is coding/non-coding</p>
<ul>
<li>Gregarious+coding: pair programming, code review, group debugging, hackathons</li>
<li>Gregarious+non-coding: standing around a whiteboard figuring out an architecture, reviewing an incident together, hashing out requirements via discussions</li>
<li>Solitary+coding: put on those headphones and make the software work. Make it beautiful. Make it sing to me.</li>
<li>Solitary+non-coding: think hard about some really tricky algorithm. Gather a bunch of written input and write a design document.</li>
</ul>
<p>Disclaimers:</p>
<ol>
<li>Unlike the classic 2×2 matrix, no quadrant is better than the others. Individual personalities, whether people happen to click, and other factors will push in various directions or towards a mix.</li>
<li>This describes various activities on a technical track. I'm not trying to describe management track.</li>
</ol>Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-35720376809602103502019-09-13T16:20:00.000-07:002019-09-16T19:01:57.623-07:00A Structured RFC Process by Phil Calçado<p>When I wrote <a href="http://jkingdon2000.blogspot.com/2019/06/technical-design-whether-who-how-and.html">Technical design: whether, who, how, and what</a> it was partly because I haven't seen a lot of guides of this sort. I'm pleased to say that Phil Calçado has offered a similar how-to at <a href="https://philcalcado.com/2018/11/19/a_structured_rfc_process.html">A Structured RFC Process</a>.</p>
<p>Some of the key similarities between <i>A Structured RFC Process</i> and my post are: (1) a discussion of who should be involved, what kinds of topics need this sort of process, what to include in such a design, and what it looks like to solicit and get feedback, (2) the focus is on the higher level or more important aspects, not specifying every detail, (3) the emphasis is on feedback and discussion, not on formal sign-offs, budgets, or other things which might be worth nailing down, but not in this sort of design, (4) relatedly, documents (and other artifacts like presentation slides or videos) produced during technical design have a relatively short half-life. They sometimes can be helpful well into the future but that's not their main purpose. Their main purpose is is to flesh out a change and organize writing code and whatever documents you have to describe "this is the current state of our system" (API documentation, wiki pages, or whatever you find useful). As Calçado says, "once an RFC moves away from Feedback requested, it is considered a historical artifact".</p>
<p>A few differences between my post and <i>A Structured RFC Process</i> are: (1) although I started thinking of a document with comments (as described in <i>A Structured RFC Process</i>), as I wrote I realized that most of what I was saying also applied to hallway conversations, presentations, or other modes of communication, (2) I include a list of technical issues you might want to address (or might not).</p>
<p>One interesting observation was "It is not uncommon for engineers to try and use the process as a way to sell an idea that hasn’t been approved by their stakeholders or managers" which I certainly have seen. Depending on how much of a power vacuum we are talking about (or, relatedly, lack of clear priorities), this could be a large or a small problem, but approaching design deliberately is not a substitute for making choices. It is at best a way to help clarify what choices the organization is facing.</p>
<p>And my favorite quote from the whole article is "The more polished a document looks, the softer and less impactful reviews tend to be". I love this. Not only does it match suggestions I've heard in other realms (for example, "to get good feedback on a user interface, show someone a napkin sketch, not a pixel-perfect mockup"), but it helps clarify one of the reasons why I've not always seen good results from highly formalized documents written in very structured and detailed ways. Not only is the <i>content</i> of such documents sometimes buried in a lot of boilerplate and irrelevance, but the very <i>form</i> discourages the kind of engagement which would make them seeds for raising issues which might be otherwise missed.</p>
Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-19171720474636521802019-06-18T08:00:00.001-07:002023-04-21T13:24:23.398-07:00Technical design: whether, who, how, and what<p><i>There is a (lightly edited) 2023 revision to this article at <a href="https://www.blogger.com/blog/post/edit/37120426/3347101560781329801">https://www.blogger.com/blog/post/edit/37120426/3347101560781329801</a> which replaces this version. <br /></i></p><p><i>This is all on one page; it was originally published 20 Aug 2018 as a four part series: <a href="http://jkingdon2000.blogspot.com/2018/08/do-i-need-technical-design.html">1</a>
<a href="http://jkingdon2000.blogspot.com/2018/08/who-drives-technical-design.html">2</a>
<a href="http://jkingdon2000.blogspot.com/2018/08/how-do-i-develop-and-promote-my.html">3</a>
<a href="http://jkingdon2000.blogspot.com/2018/08/what-goes-into-technical-design.html">4</a>
</i><br />
</p><h2>Do I need a technical design?</h2>
<br />
In agile software development, there is architecture (decisions that are hard to change) and incremental design. Architecture, in this sense, is a pretty small number of things—programming language and probably application frameworks and data storage. Incremental design is the norm: we add classes, endpoints, and database tables as we identify a need for them, or remove them as they are unneeded or replaced.<br />
<br />
But what about decisions in between these two extremes? For example, it used to be that users all signed up for the website as individuals, and now there is a need for some kind of organization which can manage the users under it. Or we used to have a bunch of separate products with their own logins, apps, and management and now there is a need to do some or all of those things in ways which apply to all products. Or our application used to assume that all users needed to be connected to the internet at all times, and now we want to build in offline operation.<br />
<br />
I won’t completely rule out handling larger changes via the usual communication of incremental development—pair programming, discussion of individual stories, pull request review, and the like. But it can be hard to maintain a clear idea of the larger design that way, and I have usually been happier with a discussion which happens at a higher level and whose goal is to get a direction into which we can fit in the smaller decisions that we will make as we go.<br />
<br />
I’ll write more later about who should drive this process, how to develop such a design, and what is worth writing down and communicating. But I’ll conclude this introductory post by asking when we should be doing this design.<br />
<br />
It is tempting to say that the high level design of a system must happen before we can start breaking down the work or implementing pieces of it. Which sounds good, and is nice when it works out, but I have yet to see a design of this sort which does not get changed during implementation. There’s a lot of reality check (interactions with existing functionality, feedback which we only get when we have an early version to show, complications which we didn’t notice at first). Therefore I wouldn’t try to finalize the design before we start acting on it. And I wouldn’t go to the other extreme—of trying to make major changes in a fully incremental way and doing all the communication after the fact. My preference is to start with rough ideas and conversations about the design, and as those get refined and conversations continue, there is a point where the general contours start falling into place. That’s about when I start implementation. I want at least some of the coding to be happening (even if we know we might be revising it later), because otherwise I don’t really trust the design. In parallel, I’m stepping up the communication (documents, meetings, etc). As things fall into place (which may include allocating people’s time, agreeing on technical or business decisions, and getting a clearer picture of implementation choices), you’ll fall into the rhythm of building the thing, because the general contours of what you are building have been established by this point.
<h2>Who drives a technical design?</h2>
<br />
So we have a problem which is meaty enough that we don’t think we want to approach it in a purely tactical way, and we’ll even assume we have defined at least the general outlines of what we want this design to accomplish. Who should turn this into a design detailed enough to implement?<br />
<br />
Before I discuss who, let me say this is an intrinsically messy process. There are a bunch of things we want out of our design. Things to do now or save for another day. People (in various roles) with opinions (either because, well, people have opinions, or more nobly, because they have a specific organizational goal they are trying to achieve). See for example Gregor Hohpe’s <a href="https://martinfowler.com/articles/architect-elevator.html" style="text-decoration: none;">The Architect Elevator</a>. Issues like reliability, security, accessibility, and branding. A large design space (a distinguishing character of software being its malleability—or at least potential for malleability). Pros and cons for pretty much every aspect.<br />
<br />
If that seems daunting, don’t despair. Just don’t be surprised if a decision which was discussed at length, carefully considered, agreed by all, and signed off subsequently starts to seem less settled. Or someone who you had thought was aware of what was going on suddenly “discovers” your design and has suggestions. Or your scope seems to keep expanding or contracting.<br />
<br />
The most important person in this process is the one who is refining the design and who will be involved in implementing it. We can call them the “<b>responsible</b>” person (although don’t think of the roles too rigidly—I did say this process tends to be on the messy side, didn’t I?). To do all these things, and have time for this design, the responsible person needs to be able to focus on this (usually, this means they aren’t a manager).<br />
<br />
But that person can’t produce a good design by sitting in a room and thinking hard (if for no other reason, because getting buy-in is a key part of what will make this design get implemented and achieve its goals). Therefore their main activity is going to be communication. I’ll have a separate post about how to communicate and what to communicate, but in the context of “who”, identify who should be “<b>consulted</b>”. That is, who needs to be aware of the design and would have good ideas about how to do it. Broadcasting what you are doing and inviting input works well, but I’d also directly seek out the people who will be most knowledgeable or important.<br />
<br />
One rule of thumb for involving a lot of people is “accept input widely, accept direction narrowly”. You want to hear from as many perspectives as you can. Whether or not you take the advice, thank people and appreciate that they took the time to engage with you. These will be the people who help communicate the changes you are making.<br />
<br />
Saying “accept direction narrowly” raises the question of who ultimately will be deciding. This role is generally called the “<b>approver</b>” and will often be the manager of the responsible person (the details will depend on your organization, though). Sign-offs are a good way of formalizing decisions already made and making sure that there is sufficient buy-in throughout the organization. They aren’t good at exploring different possible solutions or weighing pros and cons, so think of formal sign-off type processes (if you have them) as a way of ratifying what is already understood, not as a way of hashing out agreements.<br />
<br />
Lastly we have people who aren’t necessarily providing input but who should be “<b>informed</b>” about the design. The basic goal here is to cast as wide a net as feasible (in accordance with “err on the side of overcommunicating” which tends to be good advice especially in larger organizations). Think of ways to reach a variety of audiences: different levels of detail, different ways of presenting the work (for example, it can work to have one document which is technical and one which is more about the business goals and rationales—as long as they are reasonably in sync on topics such as what is in or out of scope), or different places you can announce what you are doing and offer to answer questions or sync up with interested parties.<br />
<br />
Describing the responsible, approver, consulted, and informed roles makes it clear that communication is central to the process of making technical decisions and being ready to put them into practice. The next two parts of this series will be about how to communicate, and what topics to include in that communication.
<h2>How do I develop and promote my technical design?</h2>
In the first two parts of this series we figured out we needed some kind of technical design, and we figured out who should be making that happen. How does the responsible party get this thing going? Do you call a meeting? Write something up?<br />
<br />
Typing “useless meeting” into an internet search engine and reading the results should be enough to give us pause about calling a meeting to hash out our technical design. Yet in so many organizations the meeting is the mechanism by which attention is allocated, or is otherwise necessary. So first, what are the pitfalls? The usual risk of a meeting turning into (too much of) an open ended discussion is exacerbated by the large design space and many stakeholders. Another sign that meeting discussion is a bad idea is if the wrong people are there: don’t hesitate to say “can the three of us (less than the whole meeting) have a break-out on this topic after the meeting?” or “would you be willing to talk to X (who is not present) and bring the information back?” Set your goals, such as (1) make a brief announcement about what is underway and how people can get more details or engage further, (2) present your design to date and solicit clarifying questions, or (3) give people an opportunity to raise concerns to be addressed in the future. Or if you do want a longer discussion, set the topic, keep an eye on the clock, and don’t be afraid to steer the group back to the agenda. Also, aim for a level of detail appropriate for the people in the meeting. Software developers may be most interested in database schemas and code organization, infrastructure engineers may be most interested in reliability, security or how your design is spread across various machines, product may be most interested in what functionality your design will or will not unlock, and so on.<br />
<br />
I’ve often had good luck circulating the design in document form. People have something to react to and can leave comments on the document itself or in other ways. So is this a Big Design Up Front? Not exactly. I’m aiming for something closer to a High Level Design Written As We Need It. It is at a higher level than code. It is at a higher level than detailed descriptions of functionality (click on button X and see the following fields with the following error conditions). It might contain things like database schemas or protocol specifications, although sometimes even that can be a bit fine grained.<br />
<br />
What is a design document for? First of all, as a communication tool. Secondly, to clarify the thinking of the person writing it. What about things like traceability between requirements and implementation, justifying the need for making a change, or documenting what has been changed? I would tend to think of those kinds of documents (how many you need will vary depending on your situation) as separate. The design doc is written and revised as you are thinking something through and figuring it out. More concrete documents (including breakout into tasks, specifying behaviors in detail, or explaining code details), have a greater need for detail and precision and are the output of the design process, although of course the design document can link to them as they are created. Seeing the design document as a communication tool helps focus the process of writing it. Imagine that it is a conference talk and you are trying to figure out who is the audience and what they would want to know about your design.<br />
<br />
Expect to iterate on the design. Gather some ideas. Think about them and boil them down to a proposed design. Talk to people one on one. Circulate it in writing. Figure out how else to get it out there. That will generate ideas and reactions. Figure out what to revise based on that. Expect to repeat this process until there is a sufficient degree of convergence on a course of action. Don’t fall into either the extreme of spending all your time talking to people (and not getting around to taking in what they said, researching things as needed, and making some decisions), or the other extreme, of thinking through something and coming up with something which makes sense to you, but which may lack buy-in from other people or may miss important requirements.<br />
<br />
So we are developing our design and communicating in diverse ways (presentations, written documents, informal discussions, and yes maybe even meetings). But what topics should we cover? That will be the subject of the last post of this series.
<h2>What goes into a technical design?</h2>
In the first three posts of this series we decided we need a technical design, figured out who would be doing it, and how we’ll be sending it out and getting input. But what is the content of that communication (for example, what sections would we put into a written design document)?<br />
<br />
What to include will vary depending on your organization and the needs of a particular design. For an early stage startup, anything relating to scaling and operations may take a back seat to “am I building something people want and how can I most quickly validate my hypothesis?”. For a company in a highly regulated space, there may be a lot of requirements specific to your field.<br />
<br />
The same applies to an individual design. Does my design concern a server with a high or low need to be available? Does my design concern data which is sensitive? Does this design change anything related to this topic? (If not there’s probably little to say on the subject). For that reason, I’d suggest treating templates (including this article) as guidelines, and omitting sections which don’t seem relevant. One of the fastest ways to lose an audience is to include a bunch of material that you aren’t very interested in (and probably didn’t do a very good job with). And of course to prioritize everything is to prioritize nothing, a good motto in a variety of contexts.<br />
<br />
So, what might we include?<br />
<br />
<div>
<b>Goals and non-goals</b></div>
<div>
These are perhaps the most important sections. If you can figure out what your design achieves and what you are leaving for another day or deciding is not worth doing, you are well down the path of figuring out how to do it.</div>
<div>
<b>Description of the proposed solution</b></div>
<div>
What changes will we make to code, data, networks, and hardware? How does this design achieve the goals? Give enough detail that people can see some of the implications of various choices, but try to avoid the kinds of details which can easily be fleshed out during implementation.</div>
<div>
<b>Security</b></div>
<div>
What data is stored and sent where? How is access controlled? If cryptography is involved, how are keys managed and have we chosen appropriate algorithms? Are some parts of the system isolated from others and if so how?</div>
<div>
<b>Reliability</b></div>
<div>
Is there redundancy? What are the consequences of network outages? If data is stored in a master-slave setup, how do we elect a new master? If data is written multiple places how do we reconcile them? Are there rate limits or other ways of keeping a problem one place from cascading elsewhere?</div>
<div>
<b>Capacity</b></div>
<div>
What is the expected load on the various systems involved? Does load ramp up gradually or do we expect a sudden spike in traffic? What needs to be handled manually and is there sufficient staffing to do it?</div>
<div>
<b>Monitoring</b></div>
<div>
Do we need to report new metrics? How will we know about errors?</div>
<div>
<b>Data analytics</b></div>
<div>
How will we measure usage of the new functionality? What kind of analysis might we want to do?</div>
<div>
<b>History</b></div>
<div>
Has the company considered this problem before? What previous decisions got us here? If there are documents describing previous designs, I tend to just link to them rather going into a lot of detail about what has gone before.</div>
<div>
<b>Storage</b></div>
<div>
What database(s) are involved (new or existing)? What changes in database schemas are required?</div>
<div>
<b>Interfaces between systems</b></div>
<div>
Defining these can help clarify the design and is particularly helpful if one of the functions of your design is to coordinate between different teams or companies who are responsible for different pieces.</div>
<div>
<b>Alternatives</b></div>
<div>
How else did we consider solving the problem? Why did we choose the solution we are proposing?</div>
<div>
<b>Open questions</b></div>
<div>
This section is particularly helpful if you know certain topics are controversial or warrant further discussion. As questions are resolved, move items from here into the main design section or the alternatives section.</div>
<div>
<b>Rollout</b></div>
<div>
In what order are we building this? Are we shipping it continuously? In a series of phases? Is it rolled out selectively to certain users?</div>
<br />
These questions can be taken as a template for a design document, but they also can be used to figure out who to go talk to, what to put into a presentation, or what anticipated questions to prepare for.<br />
<br />
There’s a lot in this series of blog posts about things to do: Did you talk to X? Did you consider Y? What if we did Z? And those are all very helpful up to a point. But only do those things which seem necessary for your particular organizational culture and problem you are trying to solve. The purpose of all these suggestions is to help you build things and solve problems, so as you go, don’t be afraid to keep asking yourself and others: Are people on the same page now? Is this enough specificity to build this? Is my technical design sufficient for what I need?
Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com1tag:blogger.com,1999:blog-37120426.post-12610314952662168832018-08-29T17:39:00.000-07:002018-08-29T17:47:14.988-07:00Who drives a technical design?<i>Part 2 of a 4 part series</i><i> (originally posted 20 Aug 2018)</i><br />
<br />
So we have a problem which is meaty enough that we don’t think we want to approach it in a purely tactical way, and we’ll even assume we have defined at least the general outlines of what we want this design to accomplish. Who should turn this into a design detailed enough to implement?<br />
<br />
Before I discuss who, let me say this is an intrinsically messy process. There are a bunch of things we want out of our design. Things to do now or save for another day. People (in various roles) with opinions (either because, well, people have opinions, or more nobly, because they have a specific organizational goal they are trying to achieve). See for example Gregor Hohpe’s <a href="https://martinfowler.com/articles/architect-elevator.html" style="text-decoration: none;">The Architect Elevator</a>. Issues like reliability, security, accessibility, and branding. A large design space (a distinguishing character of software being its malleability—or at least potential for malleability). Pros and cons for pretty much every aspect.<br />
<br />
If that seems daunting, don’t despair. Just don’t be surprised if a decision which was discussed at length, carefully considered, agreed by all, and signed off subsequently starts to seem less settled. Or someone who you had thought was aware of what was going on suddenly “discovers” your design and has suggestions. Or your scope seems to keep expanding or contracting.<br />
<br />
The most important person in this process is the one who is refining the design and who will be involved in implementing it. We can call them the “<b>responsible</b>” person (although don’t think of the roles too rigidly—I did say this process tends to be on the messy side, didn’t I?). To do all these things, and have time for this design, the responsible person needs to be able to focus on this (usually, this means they aren’t a manager).<br />
<br />
But that person can’t produce a good design by sitting in a room and thinking hard (if for no other reason, because getting buy-in is a key part of what will make this design get implemented and achieve its goals). Therefore their main activity is going to be communication. I’ll have a separate post about how to communicate and what to communicate, but in the context of “who”, identify who should be “<b>consulted</b>”. That is, who needs to be aware of the design and would have good ideas about how to do it. Broadcasting what you are doing and inviting input works well, but I’d also directly seek out the people who will be most knowledgeable or important.<br />
<br />
One rule of thumb for involving a lot of people is “accept input widely, accept direction narrowly”. You want to hear from as many perspectives as you can. Whether or not you take the advice, thank people and appreciate that they took the time to engage with you. These will be the people who help communicate the changes you are making.<br />
<br />
Saying “accept direction narrowly” raises the question of who ultimately will be deciding. This role is generally called the “<b>approver</b>” and will often be the manager of the responsible person (the details will depend on your organization, though). Sign-offs are a good way of formalizing decisions already made and making sure that there is sufficient buy-in throughout the organization. They aren’t good at exploring different possible solutions or weighing pros and cons, so think of formal sign-off type processes (if you have them) as a way of ratifying what is already understood, not as a way of hashing out agreements.<br />
<br />
Lastly we have people who aren’t necessarily providing input but who should be “<b>informed</b>” about the design. The basic goal here is to cast as wide a net as feasible (in accordance with “err on the side of overcommunicating” which tends to be good advice especially in larger organizations). Think of ways to reach a variety of audiences: different levels of detail, different ways of presenting the work (for example, it can work to have one document which is technical and one which is more about the business goals and rationales—as long as they are reasonably in sync on topics such as what is in or out of scope), or different places you can announce what you are doing and offer to answer questions or sync up with interested parties.<br />
<br />
Describing the responsible, approver, consulted, and informed roles makes it clear that communication is central to the process of making technical decisions and being ready to put them into practice. The next two parts of this series will be about how to communicate, and what topics to include in that communication.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-33155893817586374182018-08-29T17:34:00.000-07:002018-08-29T17:46:50.547-07:00Do I need a technical design?<i>Part 1 of a 4 part series (originally posted 20 Aug 2018)</i><br />
<br />
In agile software development, there is architecture (decisions that are hard to change) and incremental design. Architecture, in this sense, is a pretty small number of things—programming language and probably application frameworks and data storage. Incremental design is the norm: we add classes, endpoints, and database tables as we identify a need for them, or remove them as they are unneeded or replaced.<br />
<br />
But what about decisions in between these two extremes? For example, it used to be that users all signed up for the website as individuals, and now there is a need for some kind of organization which can manage the users under it. Or we used to have a bunch of separate products with their own logins, apps, and management and now there is a need to do some or all of those things in ways which apply to all products. Or our application used to assume that all users needed to be connected to the internet at all times, and now we want to build in offline operation.<br />
<br />
I won’t completely rule out handling larger changes via the usual communication of incremental development—pair programming, discussion of individual stories, pull request review, and the like. But it can be hard to maintain a clear idea of the larger design that way, and I have usually been happier with a discussion which happens at a higher level and whose goal is to get a direction into which we can fit in the smaller decisions that we will make as we go.<br />
<br />
I’ll write more later about who should drive this process, how to develop such a design, and what is worth writing down and communicating. But I’ll conclude this introductory post by asking when we should be doing this design.<br />
<br />
It is tempting to say that the high level design of a system must happen before we can start breaking down the work or implementing pieces of it. Which sounds good, and is nice when it works out, but I have yet to see a design of this sort which does not get changed during implementation. There’s a lot of reality check (interactions with existing functionality, feedback which we only get when we have an early version to show, complications which we didn’t notice at first). Therefore I wouldn’t try to finalize the design before we start acting on it. And I wouldn’t go to the other extreme—of trying to make major changes in a fully incremental way and doing all the communication after the fact. My preference is to start with rough ideas and conversations about the design, and as those get refined and conversations continue, there is a point where the general contours start falling into place. That’s about when I start implementation. I want at least some of the coding to be happening (even if we know we might be revising it later), because otherwise I don’t really trust the design. In parallel, I’m stepping up the communication (documents, meetings, etc). As things fall into place (which may include allocating people’s time, agreeing on technical or business decisions, and getting a clearer picture of implementation choices), you’ll fall into the rhythm of building the thing, because the general contours of what you are building have been established by this point.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-60927557844439499152018-08-29T17:29:00.000-07:002018-08-29T17:47:28.545-07:00How do I develop and promote my technical design?<i>This is part 3 of a 4 part series</i><i> (originally posted 20 Aug 2018)</i><br />
<br />
In the first two parts of this series we figured out we needed some kind of technical design, and we figured out who should be making that happen. How does the responsible party get this thing going? Do you call a meeting? Write something up?<br />
<br />
Typing “useless meeting” into an internet search engine and reading the results should be enough to give us pause about calling a meeting to hash out our technical design. Yet in so many organizations the meeting is the mechanism by which attention is allocated, or is otherwise necessary. So first, what are the pitfalls? The usual risk of a meeting turning into (too much of) an open ended discussion is exacerbated by the large design space and many stakeholders. Another sign that meeting discussion is a bad idea is if the wrong people are there: don’t hesitate to say “can the three of us (less than the whole meeting) have a break-out on this topic after the meeting?” or “would you be willing to talk to X (who is not present) and bring the information back?” Set your goals, such as (1) make a brief announcement about what is underway and how people can get more details or engage further, (2) present your design to date and solicit clarifying questions, or (3) give people an opportunity to raise concerns to be addressed in the future. Or if you do want a longer discussion, set the topic, keep an eye on the clock, and don’t be afraid to steer the group back to the agenda. Also, aim for a level of detail appropriate for the people in the meeting. Software developers may be most interested in database schemas and code organization, infrastructure engineers may be most interested in reliability, security or how your design is spread across various machines, product may be most interested in what functionality your design will or will not unlock, and so on.<br />
<br />
I’ve often had good luck circulating the design in document form. People have something to react to and can leave comments on the document itself or in other ways. So is this a Big Design Up Front? Not exactly. I’m aiming for something closer to a High Level Design Written As We Need It. It is at a higher level than code. It is at a higher level than detailed descriptions of functionality (click on button X and see the following fields with the following error conditions). It might contain things like database schemas or protocol specifications, although sometimes even that can be a bit fine grained.<br />
<br />
What is a design document for? First of all, as a communication tool. Secondly, to clarify the thinking of the person writing it. What about things like traceability between requirements and implementation, justifying the need for making a change, or documenting what has been changed? I would tend to think of those kinds of documents (how many you need will vary depending on your situation) as separate. The design doc is written and revised as you are thinking something through and figuring it out. More concrete documents (including breakout into tasks, specifying behaviors in detail, or explaining code details), have a greater need for detail and precision and are the output of the design process, although of course the design document can link to them as they are created. Seeing the design document as a communication tool helps focus the process of writing it. Imagine that it is a conference talk and you are trying to figure out who is the audience and what they would want to know about your design.<br />
<br />
Expect to iterate on the design. Gather some ideas. Think about them and boil them down to a proposed design. Talk to people one on one. Circulate it in writing. Figure out how else to get it out there. That will generate ideas and reactions. Figure out what to revise based on that. Expect to repeat this process until there is a sufficient degree of convergence on a course of action. Don’t fall into either the extreme of spending all your time talking to people (and not getting around to taking in what they said, researching things as needed, and making some decisions), or the other extreme, of thinking through something and coming up with something which makes sense to you, but which may lack buy-in from other people or may miss important requirements.<br />
<br />
So we are developing our design and communicating in diverse ways (presentations, written documents, informal discussions, and yes maybe even meetings). But what topics should we cover? That will be the subject of the last post of this series.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-23825670596429033022018-08-28T16:49:00.000-07:002018-08-29T17:47:39.678-07:00What goes into a technical design?<i>Part 4 of a 4 part series</i><i> (originally posted 20 Aug 2018)</i><br />
<br />
In the first three posts of this series we decided we need a technical design, figured out who would be doing it, and how we’ll be sending it out and getting input. But what is the content of that communication (for example, what sections would we put into a written design document)?<br />
<br />
What to include will vary depending on your organization and the needs of a particular design. For an early stage startup, anything relating to scaling and operations may take a back seat to “am I building something people want and how can I most quickly validate my hypothesis?”. For a company in a highly regulated space, there may be a lot of requirements specific to your field.<br />
<br />
The same applies to an individual design. Does my design concern a server with a high or low need to be available? Does my design concern data which is sensitive? Does this design change anything related to this topic? (If not there’s probably little to say on the subject). For that reason, I’d suggest treating templates (including this article) as guidelines, and omitting sections which don’t seem relevant. One of the fastest ways to lose an audience is to include a bunch of material that you aren’t very interested in (and probably didn’t do a very good job with). And of course to prioritize everything is to prioritize nothing, a good motto in a variety of contexts.<br />
<br />
So, what might we include?<br />
<br />
<div>
<b>Goals and non-goals</b></div>
<div>
These are perhaps the most important sections. If you can figure out what your design achieves and what you are leaving for another day or deciding is not worth doing, you are well down the path of figuring out how to do it.</div>
<div>
<b>Description of the proposed solution</b></div>
<div>
What changes will we make to code, data, networks, and hardware? How does this design achieve the goals? Give enough detail that people can see some of the implications of various choices, but try to avoid the kinds of details which can easily be fleshed out during implementation.</div>
<div>
<b>Security</b></div>
<div>
What data is stored and sent where? How is access controlled? If cryptography is involved, how are keys managed and have we chosen appropriate algorithms? Are some parts of the system isolated from others and if so how?</div>
<div>
<b>Reliability</b></div>
<div>
Is there redundancy? What are the consequences of network outages? If data is stored in a master-slave setup, how do we elect a new master? If data is written multiple places how do we reconcile them? Are there rate limits or other ways of keeping a problem one place from cascading elsewhere?</div>
<div>
<b>Capacity</b></div>
<div>
What is the expected load on the various systems involved? Does load ramp up gradually or do we expect a sudden spike in traffic? What needs to be handled manually and is there sufficient staffing to do it?</div>
<div>
<b>Monitoring</b></div>
<div>
Do we need to report new metrics? How will we know about errors?</div>
<div>
<b>Data analytics</b></div>
<div>
How will we measure usage of the new functionality? What kind of analysis might we want to do?</div>
<div>
<b>History</b></div>
<div>
Has the company considered this problem before? What previous decisions got us here? If there are documents describing previous designs, I tend to just link to them rather going into a lot of detail about what has gone before.</div>
<div>
<b>Storage</b></div>
<div>
What database(s) are involved (new or existing)? What changes in database schemas are required?</div>
<div>
<b>Interfaces between systems</b></div>
<div>
Defining these can help clarify the design and is particularly helpful if one of the functions of your design is to coordinate between different teams or companies who are responsible for different pieces.</div>
<div>
<b>Alternatives</b></div>
<div>
How else did we consider solving the problem? Why did we choose the solution we are proposing?</div>
<div>
<b>Open questions</b></div>
<div>
This section is particularly helpful if you know certain topics are controversial or warrant further discussion. As questions are resolved, move items from here into the main design section or the alternatives section.</div>
<div>
<b>Rollout</b></div>
<div>
In what order are we building this? Are we shipping it continuously? In a series of phases? Is it rolled out selectively to certain users?</div>
<br />
These questions can be taken as a template for a design document, but they also can be used to figure out who to go talk to, what to put into a presentation, or what anticipated questions to prepare for.<br />
<br />
There’s a lot in this series of blog posts about things to do: Did you talk to X? Did you consider Y? What if we did Z? And those are all very helpful up to a point. But only do those things which seem necessary for your particular organizational culture and problem you are trying to solve. The purpose of all these suggestions is to help you build things and solve problems, so as you go, don’t be afraid to keep asking yourself and others: Are people on the same page now? Is this enough specificity to build this? Is my technical design sufficient for what I need?
Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-88603043163738171432017-01-28T15:43:00.000-08:002017-01-28T15:43:34.706-08:00Can you pair program at a company where pair programming isn't done?I've been hooked on pair programming from the time I first tried it. I love pairing as a way to transfer knowledge (either about technology or about our product), build motivation, and build teams. Software development is a long series of decisions both large and small, many of which could plausibly go another way. When I'm soloing it is so much easier for me to get stuck on any one of them.<br />
<br />
If you are in a company where pairing is the norm, you'll do it, but what if people are just curious about pairing? Or willing to try but who don't know much about it? Here's what had worked for me. First of all, I invite people to pair for 1.5 hour blocks, usually scheduled on our calendars (shorter can work but to go longer (a) requires a break in the middle, and (b) requires more buy-in from my pair than I sometimes have). Secondly, when I'm asking a co-worker to pair I ask them to pair on a specific task which I am up to speed on (for example, which has been assigned to me). Ideally, the task also requires knowledge they have that I don't (familiarity with a particular part of the codebase for example).
During the pairing I apply pairing skills I've learned over the years (for example, handing the keyboard to a bored pair or saying "let's give it a try and see what happens" rather than "that won't work"). I wrap up by the end of the scheduled time (continuing after a break if both people are psyched is an option but usually 1.5 hours is quite enough for people who aren't in the pairing habit). As we wrap up, I make sure to thank them and tell them how helpful it was (this is usually quite sincere - I did mention that I go faster when pairing than soloing, didn't I?). If the task isn't done, I usually finish it up soloing (especially if the remaining items are fairly straightforward once pairing makes some of the bigger decisions).<br />
<br />
Afterwards, I tell others, for example in a retrospective or a 1:1 with my manager, how much I enjoyed pairing and/or concrete benefits like "we were able to work out the interface between these two components much more easily than if I had been soloing on one side and you had been soloing on the other". The goal here is not to tell people they have to pair, the goal is to make it feel like they are missing out on something great if they don't.<br />
<br />
Pairing got one of my teams out of a sticky trap. There was a section of the code which only one person understood. We saw this was a problem and the person who knew the code wanted to share his knowledge. For our first attempt, he explained it in a conference room with a whiteboard and a projector. Perhaps that helped a bit, but the explanation didn't made as much sense to the audience as to the presenter and we adjourned with confusion and frustration, or at least with limited comprehension. Later I had reason to do something to that code, and so I asked the expert whether he would pair on it. Mechanically, it was miserable. We didn't share a fluent spoken language and he used a customized setup (using vi and virtual machines) which meant that I mostly watched him type or told him what to type. A far cry from the easy flow between two people which sold me on pairing in the first place! But guess what? I learned a whole lot more about that code than I did from sitting in a conference room. Other people started working on that code and the person who had been the expert could get help and feel less alone. Here I started to formulate my belief that even a very small amount of pairing was better than none at all.<br />
<br />
In my other example the surprise was even more pleasant and delayed. We were in a company where pairing was often mentioned, sometimes practiced (at least in some teams or situations), but was certainly optional and not part of most people's habits on a regular basis. One of the people on my team was nice but also seemed like a loner: often wearing headphones, not speaking up much in meetings, and getting a lot done but in a heads-down kind of way. Not my first choice for someone to ask to pair. But in a few cases, I carefully came up with a focused, suitable task and asked him to pair. We paired maybe half a dozen times (if that) over a one year period. It energized me and I appreciated his willingness to put up with my eccentric desire to pair. Fast forward a year or so, we now work for different companies, and he tells me that pairing with me was one of the highlights of his entire two year time at the company! I was floored. I knew I enjoyed working with him in general but I was completely unaware of what he was getting out of pairing.<br />
<br />
Do I recommend being a pairing pioneer? Well, it isn't always easy and to be perfectly honest, my current job search is for a situation where pairing is already more established and common. But if you like pairing and find yourself in a non-pairing or low-pairing situation? Sure, give it a shot. As long as people approach it with an open mind (on both sides), the only thing you are risking is 1.5 hours of your time.
Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-6257009178937779292013-11-23T11:40:00.000-08:002013-11-23T11:47:56.825-08:00Securing package distribution with TUF<p>Suppose you are downloading a new fun game for your computer and you want to know whether it is going to do what it claims (clicking on cows, let's say) or whether it is going to send all your data (credit card numbers you type, let's say) to a shadowy cabal in Martha's Vineyard or Napa Valley or whereever shadowy cabals are found these days. For the sake of argument, let's say that you have heard good things about the a (hypothetical) open source project called FreeCowClicker2 written by Ilia Bogomips and you want to try it out.</p>
<p>Well, in some cases the authors of FreeCowClicker2 might run a download site and you might get it there, but most of the time you'll probably be getting it from a package repository, such as a linux distribution, a programming-language-specific repository such as CPAN (perl), PyPI (python), rubygems (ruby), or something like addons.mozilla.org. How do I know I'm getting the package I want, if (a) I am connecting to a potentially dodgy WiFi access point or there is some other way in which the shadowy cabal has gotten into my network, or (b) one of the servers involved in serving up the files, or mirroring them, is under the control of our shadowy cabal?</p>
<p>If you are a little bit familiar with this stuff, you are probably saying "signed packages", as found in for example <a href="https://fedoraproject.org/keys" >Fedora</a> or <a href="https://wiki.debian.org/SecureApt" >Debian</a>. And that indeed is what I'm getting at, specifically <a href="http://theupdateframework.com/" >TUF</a> (The Update Framework). TUF aims to be usable by any package repository, but the most effort to date has been to using it for PyPI.</p>
<p>As part of <a href="https://corner.squareup.com/">Square</a>'s hack week which just concluded, a number of us looked into using TUF with rubygems, and wrote <a href="https://github.com/rubygems/rubygems/pull/719" >some code</a> to that end. Hopefully that code helps clarify what this is all about, and there is a fair bit of documentation on the TUF site, so I'll just mention a few of the high points and interesting details:</p>
<ul>
<li>TUF can upgrade its keys. Your package installer might find there are new keys (signed by the old keys) and switch to them.</li>
<li>There are multiple keys for different things. Anyone who contributes a package can have a key which is just good for signing that one package, there are separate keys which are used to say what packages were released as of a given time, and there are keys which are just used to sign the frequently used keys. Some of the keys can be kept offline, and only used maybe a few times a year.</li>
<li>TUF is fairly easy to work with. The public keys and signatures and such are kept in JSON, which means you can parse them with, say, ruby or <a href="http://stedolan.github.io/jq/">jq</a>.</li>
<li>One little example of something they thought of: when you get a reference to another TUF file you also get a signed length. That way an attacker can't substitute a multi-gigabyte file and cause a denial of service by tying up your network or computer. You just need to download as many bytes as the signature told you to, and can quit after that.</li>
</ul>
<p>There's plenty left to do to finish the job of getting rubygems to use TUF, so go look at the <a href="https://github.com/rubygems/rubygems/pull/719" >pull request</a> and start pitching in if you feel so inclined. Based on a week of looking at it, TUF does look like a solid basis for a more secure rubygems which preserves all the cool things about rubygems like letting people release gems often, letting anyone author a gem, etc. Likewise, it also seems promising for other package repositories.</p>
Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-43105861052044748902012-07-20T12:56:00.000-07:002012-10-23T06:18:17.849-07:00Parsing large XML filesEvery once in a while, we need to parse large XML files. Here "large" means that the file won't fit in memory, so we can't just suck it in using nokogiri (or our favorite in-memory XML library). SAX is fine as a low level parser to hand tell you where the tags start and end, but trying to do any significant processing will turn into spaghetti unless you have a bit of a framework. The <a href="http://jkingdon2000.blogspot.com/2008/05/parse-xml-with-saxophone.html">last time I visited this topic</a>, I ended up writing a library, saxophone, which invoked callbacks when it encountered certain named tags. Saxophone is sitting in an obscure git repository; I could put it up as a gem if someone wants it; the big question is whether there is something better out there. The wasabi WSDL parser has been trying their own mini-framework (partially special purpose) described at <a href="https://github.com/savonrb/wasabi/issues/7#issuecomment-7132552">this issue</a>.
But probably the best I've seen so far is <a href="https://github.com/ezkl/sax-machine">sax-machine</a> (specifically, the lazy option thereto). I haven't spent much time playing with it (at least not yet), but it seems like a better starting point than starting from scratch with a new gem.
If you do end up writing code directly on top of SAX, just remember this: keep a stack of start tags and end tags. Following this idiom might cut down on the buggy spaghetti that I've seen when I've tried to do without something like saxophone or sax-machine.
<b>Update</b>: I fixed the above link to the <a href="https://github.com/savonrb/wasabi">wasabi</a> issue, which had changed. Not sure how long-lived any of these links are going to be, but here's another one: <a href="https://github.com/savonrb/wasabi/blob/sax-parser/lib/wasabi/sax_parser.rb">lib/wasabi/sax_parser.rb</a> from the sax-parser branch. The key is the stack (pushed on start tag, popped on end tag) and the matchers.
Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-3196026684586557562011-12-07T20:32:00.001-08:002011-12-07T20:37:38.742-08:00count.countSometimes you pick a programming idiom because it is what you are familiar with, because you think it is expected, or because it expresses clearly what the code you are writing is trying to do. Other times, it is just too hard to resist. Lately at work at least two of us have seen <code>.count.count</code> in our rails3 code, and at first were sure it must be a typo. The real story is more fun than that, see <a href="http://nerdfeed.needfeed.com/blog/2011/12/count-count/">the nerdfeed blog</a> for more.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-92102963504401950512011-07-28T08:45:00.000-07:002011-07-28T09:13:31.191-07:00Using active record in rails migrationsMost rails developers have probably sooner or later run into the problem: if your migrations refer to active record classes and the active record classes change out from under the migration, old migrations won't work as desired any more. Whether this is a big problem or a minor annoyance depends on how often you run migrations, how many databases you have (typically one for each developer and one or more you deploy to), etc, but I've seen the problem even over the course of three developer machines and a day or two, as some refactoring made people unable to update their code and then run a only-slightly-older migration.<br /><br />One solution, advocated in the "Data migrations" section of <a href="http://robots.thoughtbot.com/post/8135270582/code-review-ruby-and-rails-idioms" >Code review: Ruby and Rails idioms</a> is just to fall back to writing migrations in SQL, bypassing active record (with the exception of the low-level parts of active record which connect to the database). This has two problems. The first is that active record doesn't help you a lot with this kind of low-level SQL construction. The example in that block post uses string interpolation to construct SQL, which they can get away with in that example (because the columns are integers) but which blows up as soon as the quoting isn't correctly handled (in a migration, this is probably just a bug rather than a security hole, but search "SQL injection" if you are unfamiliar with the problems). The second problem is that active record just is a more expressive way to manipulate data. How many people use script/console rather than script/dbconsole to look around the database?<br /><br />My recommended solution, also advocated in <a href="http://gem-session.com/2010/03/how-to-use-models-in-your-migrations-without-killing-kittens" >How to use models in your migrations (without killing kittens)</a>, is to define the classes within the migration. There's an example in that blog post, but the short summary is that if, for example, your migration wants to refer to Vendor, you put "class Vendor < ActiveRecord::Base; end" within the migration class. In some cases you might need to define a few has_many or belongs_to relationships (make sure to set class_name to refer to the migration-specific class), but the interesting (and surprising to me) thing is that I've found that in practice you don't need a whole lot of them. Just to give a few examples of what this gets you, think of things like calling find_or_create_by_name to skip creating a record if it already exists, or looking up an object by name and then using its ID in a subsequent SQL statement. If you are thinking "but I can do that in SQL", then I'm not sure I should try to convince you. But if you are thinking "yeah, that is easier / more-concise / more-readable in active record" then defining your classes in the migration gets you both this, and also lets you run migrations even after your code has continued to evolve.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-42640091270692428062011-06-28T03:00:00.000-07:002011-06-28T03:01:23.476-07:00Celebrate tau day with a few proofsWhat better way to celebrate <a href="http://tauday.com/" >Tau Day</a> than by trying your hand at writing a few proofs? Wikiproofs is a wiki which anyone can edit, and the goal is to build a library of proofs. They are written in a formal language, so the web site can check their correctness, and that is what makes it good for the tau day exercise: you get feedback about whether your proof is correct from the site as you go. The tau day exercises are intended to be of modest length and difficulty (so you can get them done on tau day) and explain everything you need to know about wikiproofs. Go to <a href="http://wikiproofs.org/wiki/index.php?title=Help:Tau_day" >Wikiproofs tau day</a> to play.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-22866713329459730282011-06-17T19:52:00.000-07:002011-06-17T19:54:15.705-07:00Help me proofread tau day exercisesIn honor of <a href="http://tauday.com" >tau day</a> (a math holiday celebrated on June 28 every year), I have written some exercises at <a href="http://wikiproofs.org/wiki/index.php?title=Help:Tau_day" >wikiproofs.org tau day</a>. If you have a little time to go to that page and try to go through the exercises, I'd appreciate your feedback on writing style, whether they are too easy or too hard, whether they were too long or too short, and any other suggestions. I'm hoping this will be a fun game/exercise for anyone interested in math but I am trying to make it accessible to someone with no experience in formal proofs.<br /><br />If you are here to read about programming, I intend to keep writing about that too, but the math proofs have been a good part of my hobby activities lately. It is kind of like programming anyway (processed by a computer, needs to follow pretty specific rules to work, can be addictive in some of the same ways).Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com2tag:blogger.com,1999:blog-37120426.post-3124148251670489312011-05-15T05:21:00.000-07:002011-05-20T06:08:23.069-07:00Scripting math proofs with RHilbertFormal math projects like <a href="http://wikiproofs.org/">wikiproofs</a> prove mathematical theorems in a way that a computer can verify. There could be several motivations for this, including finding/preventing errors in proofs, helping learners to understand a proof, or exploring the consequences of assuming a different set of axioms.<br /><br />Wikiproofs and related projects like metamath require that the person writing the proof spell it out pretty explicitly. For example, if you have 1 + 2 = 3 and you need 2 + 1 = 3, you'll need to explicitly transform one into the other. Other proof systems, like coq and isabelle, have a fairly powerful prover which can notice that you have 1 + 2 = 3 and a + b = b + a and combine those to prove 2 + 1 = 3.<br /><br />Enough background. What I've been playing with lately is a project I just started and which I am calling <a href="http://github.com/jkingdon/hilbert">Hilbert</a>. This is a marriage of a metamath-like proof engine (in this case <a href="http://github.com/TheCount/hilbert-kernel">hilbert-kernel</a>) and a generic scripting language (in this case Ruby). Writing a full prover in Ruby is of course one direction this could eventually head, but I was thinking more in terms of simpler kinds of automation (perhaps there could be a routine called "commute as needed" which would be able to turn "1 + 4 = 3 + 2" into "4 + 1 = 3 + 2" by noticing the left hand side needs to be flipped and the right hand side doesn't). I'm hoping this system will be easy for people who find Ruby more comfortable than coq or isabelle (I might count myself among them, although of course I reserve the right to learn coq or isabelle some day). It also may have other benefits, like making it easier to develop hilbert-kernel itself (for example by running hilbert-kernel tests).<br /><br /><span style="font-weight:bold;">Update 20 May 2011:</span> this is under active development, but the change which needed an update above is that I renamed the project from RHilbert to hilbert.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-58427406042528189582011-03-22T13:40:00.000-07:002011-03-22T13:54:14.575-07:00Ruby rescue gotchaToday in the codebase at my day job, I found a particularly cute bug related to "rescue" in Ruby. This isn't a particularly unknown gotcha–I've read about it on the net at least once–but this is a particularly sweet (or devious) manifestation, and as far as I can tell it was purely accidental, not a contrived example.<br /><br />The following example uses rspec, but the key thing is <code>begin; 2 / 0 rescue NoMethodError; end</code> versus <code>begin; 2 / 0; rescue NoMethodError; end</code>–what is the difference between these two statements?<br /><br /><pre><br />require File.expand_path('spec_helper', File.dirname(__FILE__))<br /><br />describe "enigma" do<br /> it "fails, but why" do<br /> lambda { begin; 2 / 0 rescue NoMethodError; end }.<br /> should raise_error(ZeroDivisionError)<br /> end<br /><br /> it "passes, but why" do<br /> lambda { begin; 2 / 0; rescue NoMethodError; end }.<br /> should raise_error(ZeroDivisionError)<br /> end<br />end<br /></pre><br /><br />I'll post the answer in a comment in a few days; feel free to post your answers as comments if you wish.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com2tag:blogger.com,1999:blog-37120426.post-68887228250293313082011-03-03T17:48:00.000-08:002011-03-03T17:54:21.452-08:00Levitation-perl and deleted filesEver since I <a href="http://jkingdon2000.blogspot.com/2011/01/tracking-mediawiki-wiki-in-git.html" >started using levitation-perl</a>, I was curious about how it handles deleted files. Well, I found out (the hard way). The symptom was that I did a routine merge and a lot of files showed up as conflicts (including ones which hadn't been edited, on the wiki or on the git side, for a long time). This is usually a symptom of git not being able to find a reasonable common ancestor.<br /><br />Turned out that any files which had been deleted in mediawiki caused what amounts to a rewrite of the git history (as if those files had never been created). This is not specific to mediawiki/levitation; the same kind of thing would happen in pure git if you ran git filter-branch or similar mechanisms to delete a file from git and make sure it was erased from the history (see for example <a href="http://progit.org/book/ch6-4.html" >Rewriting History</a> or <a href="http://lwn.net/Articles/421680/" >The trouble with firmware</a>).<br /><br />The consequence of the rewritten history for my merge was that the common ancestor was very old (prior to when the deleted file was first created, about two years ago in my case), rather than a few days old as would be the case without the history rewrite.<br /><br />How did I get my merge done? In my specific case, the content of the deleted files was nothing sensitive, so I was fine with having them remain in the git history. If they needed to stay gone from the history, I would probably have needed to follow RECOVERING FROM UPSTREAM REBASE in the git-rebase manpage.<br /><br />My solution was to create a merge commit whose parents are the two corresponding commits in the rewritten and non-rewritten history. Since the git repository I was using is public, you can <a href="http://github.com/jkingdon/wikiproofs" >follow along</a>. The commands here are somewhat edited (I've snipped out my dead ends and multiple invocations of gitk to see what I had at each step).<br /><br /><pre><br />[browse gitk history to find that 681b7936 is one of the commits<br />with message "add domain and Domain"]<br />$ git checkout 681b793629d88729b919f40d0862884147db0d8d<br />Note: checking out '681b793629d88729b919f40d0862884147db0d8d'.<br /><br />You are in 'detached HEAD' state. . . .<br />HEAD is now at 681b793... add domain and Domain<br />$ git checkout -b withdeletedfiles<br />Switched to a new branch 'withdeletedfiles'<br />$ gitk levitation/master&<br />[browse history to find that a6d0885... is the commit with message<br />"add domain and Domain"]<br />$ git checkout a6d0885<br />Note: checking out 'a6d0885'.<br /><br />You are in 'detached HEAD' state. . . .<br /><br />HEAD is now at a6d0885... add domain and Domain<br />$ git checkout -b withoutdeletedfiles<br />Switched to a new branch 'withoutdeletedfiles'<br />$ git merge -s ours withdeletedfiles<br />Merge made by ours.<br />$ git diff --stat -w withdeletedfiles withoutdeletedfiles<br /> Main/W/P/.3a/WP:INDEX | 2 --<br /> Wikiproofs/S/u/b/Subject index | 30 ------------------------------<br /> 2 files changed, 0 insertions(+), 32 deletions(-)<br />$ git checkout master<br />Switched to branch 'master'<br />$ git merge withoutdeletedfiles<br />Removing Main/W/P/.3a/WP:INDEX<br />Removing Wikiproofs/S/u/b/Subject index<br />Merge made by recursive.<br /> Main/W/P/.3a/WP:INDEX | 2 --<br /> Wikiproofs/S/u/b/Subject index | 30 ------------------------------<br /> 2 files changed, 0 insertions(+), 32 deletions(-)<br /> delete mode 100644 Main/W/P/.3a/WP:INDEX<br /> delete mode 100644 Wikiproofs/S/u/b/Subject index<br />[1]+ Done gitk<br />$ gitk&<br />[1] 3233<br />$ git show 01a25538177dbe768e130aa94f7d49be11a63733<br />commit 01a25538177dbe768e130aa94f7d49be11a63733<br />Merge: 4ba316c e908dc8<br />Author: Jim Kingdon <jkingdon@localhost.localdomain><br />Date: Tue Mar 1 20:40:57 2011 -0500<br /><br /> Merge branch 'withoutdeletedfiles'<br /><br />$ git diff --stat -w e908dc8..master<br /> Interface/S/e/t/Set theory | 13 +-<br /> Main/O/u/t/Out lines | 311 +++++++++++<br />[and the rest of the diffs I'd expect from the wiki to git]<br /> 22 files changed, 4626 insertions(+), 47 deletions(-)<br />$ git diff --stat -w 4ba316c..master<br /> Main/W/P/.3a/WP:INDEX | 2 --<br /> Wikiproofs/S/u/b/Subject index | 30 ------------------------------<br /> 2 files changed, 0 insertions(+), 32 deletions(-)<br />$ git push<br />$ <br /></pre>Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-47523281562157877972011-02-26T04:35:00.000-08:002011-02-26T04:45:49.728-08:00Four hour agile subcommittee meetingDon't know whether to file this under "<a href="http://martinfowler.com/bliki/SemanticDiffusion.html" >what happens when something becomes a buzzword</a>", "people who don't get it", or "baby steps, baby steps", but today at standup (for an organization not traditionally agile, but whose avowed intention is to become agile), someone casually mentioned the words "four hour agile subcommittee meeting". It is one of the most wonderfully oxymoronic phrases I've heard in a while.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0tag:blogger.com,1999:blog-37120426.post-30238332312430555392011-01-16T07:13:00.000-08:002011-01-16T07:29:40.520-08:00Tracking a mediawiki wiki in gitThere has been a lot of interest lately in git-backed wikis, like <a href="https://github.com/github/gollum" >gollum</a>, <a href="https://github.com/sr/git-wiki" >git-wiki</a>, or <a href="http://gitit.johnmacfarlane.net/" >gitit</a>. It is appealing to both make it easy for casual editors to make a change (without cloning anything, installing git software, having to push, etc), but also provide a git repository for things like browsing and editing the content offline, being able to merge changes, and the like. The appeal strikes me as particularly strong if the content of the wiki is software (as is envisaged for <a href="http://www.wheatfarm.org/" >wheat</a>) or something software-like, such as mathematical proofs (as in <a href="http://www.wikiproofs.org" >wikiproofs</a>).<br /><br />So a wiki which is backed by git is cool, but what if the wiki is using <a href="http://mediawiki.org/" >mediawiki</a> (perhaps the most popular wiki software, in use at wikipedia and other sites)? One might want the various features and extensions of mediawiki, or one might be looking to git-ize a wiki which has been around longer than this whole git-based wiki trend got going. One solution I've been playing with is<br /><a href="http://github.com/sbober/levitation-perl" >levitation-perl</a>.<br /><br />You first of all get an XML dump file from the wiki (most mediawikis produce these once a day and make them available for download). Then you download levitation-perl and install perl and some packages from<br />CPAN (as documented in levitation-perl's README and slightly elaborated at <a href="http://github.com/jkingdon/levitation-perl" >my fork</a>). Then you can run levitation-perl (as described in the README, again) to import all the mediawiki changes into an empty git repository. Then you can push this git repository someplace public, clone it locally, and do all the good git stuff you are used to.<br /><br />At least for me, moving changes from git to the wiki is manual, although that is perhaps in part a function of how wikiproofs works–the changes are checked for correctness as you edit the pages on the wiki, and I haven't yet bothered to try to install the correctness-checking software locally.<br /><br />When the wiki changes and you want to get all those changes into git, get a new XML dump and run levitation-perl again. You don't need to run it in an empty git repository this time–I just run it in the git repository that I used for the previous levitation-perl run (on the other hand, when I want to check out files, I clone this repository elsewhere).<br /><br />Levitation-perl hasn't gotten much attention lately, I think due to technical problems using it for something as large as wikipedia (and perhaps policy questions of what to use it for on wikipedia). But none of that really affects my use of it on wikiproofs, because wikiproofs is way smaller than wikipedia, and because my goals are mostly just wanting a way to work on wikiproofs when I don't have good internet<br />connectivity.Jim Kingdonhttp://www.blogger.com/profile/01857308320156877253noreply@blogger.com0