Joshuami2023-10-30T17:16:44+00:00http://joshuami.comJosh MitchellAccess Control Strategies for Enterprise Drupal Websites2023-10-27T00:00:00+00:00http://joshuami.com/2023/10/27/access-control-strategies-for-enterprise-drupal-websites<p>When working with large organizations—whether in private enterprise, governmental organizations, or nonprofits—editorial responsibility is often spread across multiple sub-organizations. Depending on the type of organization, those subdivisions can look very different:</p>
<ul>
<li>Department > division > section or program</li>
<li>Bureau > program > project or team</li>
<li>Brand > product line > product</li>
</ul>
<p>In all of the above structures, there is some hierarchy and some cross-organizational oversight to the editorial process combined with the overall administration of the website.</p>
<p>Organizational structure for a site can also significantly differ for publicly-accessed content versus employee intranets versus membership-based websites.</p>
<p>Drupal.org has somewhat-outdated documentation for a Comparison and Overview of Access Control modules. The documentation in question was created for Drupal 7, updated somewhat for Drupal 8—it hasn’t been maintained for Drupal 9 and 10.</p>
<p>While this overview outlines a long list of access control modules, it doesn’t speak much to the “<em>why”</em> or “<em>how does this module work”</em>. With this post, I hope to outline some approaches to access control in Drupal and talk about why you might want to take one approach over another.</p>
<p>We’ll cover:</p>
<ul>
<li>What’s in Drupal core to handle access control?</li>
<li>The simplest approach: trust more and verify</li>
<li>The Group module</li>
<li>Workbench suite of modules</li>
<li>Taxonomy Access Control Lite</li>
<li>What about Organic Groups (OG)?</li>
</ul>
<h2 id="whats-in-drupal-core-to-handle-access-control">What’s in Drupal core to handle access control?</h2>
<p>One of Drupal’s most powerful features is its roles and permissions architecture. Out of the box, you have anonymous, authenticated, and administrator roles. You can then create any named role you site needs. While adjusting permissions for a role can become a bit burdensome at scale—due to the enormous number of checkboxes on the administer permissions page—the power of setting these permissions from a single administrative interface is hard to deny.</p>
<p>For each content type created in Drupal, there are permissions associated with creating, editing, and deleting that content type. If revisions are enabled, you can further add permissions for the ability to view, revert, and delete these revisions. If workflows are enabled, you can grant the ability to move content from one moderation state to the next.</p>
<p>For sites with a limited number of roles and a clear permissions model, Drupal core can be more than sufficient to keep the editorial process in order.</p>
<h2 id="the-simplest-approach-trust-but-verify">The simplest approach: trust but verify</h2>
<p>This leads us to the simplest approach to an enterprise-ready content access model that I call “trust and verify”.</p>
<p>Often organizations will assume they need robust access control to prevent employees from doing what they shouldn’t:</p>
<ul>
<li>deleting someone else’s content</li>
<li>changing someone else’s content</li>
<li>adding content they shouldn’t</li>
<li>putting content in the wrong organizational structure</li>
</ul>
<p>While all of these could be valid issues in organizations with hundreds of editors, the question should be asked, “why did you give these people access if you can’t trust them to do their job?”</p>
<p>It seems simple, doesn’t it?</p>
<p>Still, there are many managers and executives that want a bit more assurance. Even out of the box, without additional modules, you can configure Drupal so that it requires a bit of verification for the most destructive actions.</p>
<h3 id="status-versus-state">Status versus state</h3>
<p>All content in Drupal has the option for a published status. This status is either published or not (unpublished) with default permissions that do not allow anonymous public users to see the content that is not published.</p>
<p>With core’s Content Moderation module, this can be expanded to add additional states that can be published or unpublished and default or not. The default state is referring to the publish status presented to users when they visit the canonical URL for the content.</p>
<p>While most developers I’ve met usually pick up on the concept of <em>default state</em> pretty quickly, I’ve found many editors are a bit confused that “published” and “unpublished” are both a status (on/off or true/false) as well as a moderation state (step in the workflow).</p>
<h3 id="the-recipe-for-verifying-editor-actions">The recipe for verifying editor actions</h3>
<h4 id="moderation-states">Moderation states</h4>
<p>Moderation states are steps in a workflow that an editor can use to convey the intent of the state of the revision being saved.</p>
<p>The following are a good starting point for a <em>trust but verify</em> permission model:</p>
<ul>
<li><strong>Draft</strong> - unpublished revision that is not a default current state</li>
<li><strong>In review</strong> - unpublished revision (not default) with the intent that content in this state should be reviewed by someone with the authority to approve the revision and publish it</li>
<li><strong>Published</strong> - published default revision that can be viewed by anonymous users</li>
<li><strong>Unpublished</strong> - unpublished default revision that removes visibility to the content</li>
<li><strong>Marked for deletion</strong> (optional) - unpublished default revision for identifying content that is intended for deletion that can be reviewed by a more advanced role for removal from the system.</li>
</ul>
<p>It is important that your editors understand the meaning of “revision” and that revisions can be unpublished. Draft, in review, unpublished, and marked for deletion are all unpublished revisions that the public cannot access. If a user without the permission to see these revisions visits the URL for the content, they will get an access denied error by default.</p>
<h4 id="roles">Roles</h4>
<p>Roles in Drupal are assigned permission. Role assignments are additive. To put that another way, your permissions are the sum of all the roles to which you are assigned. While this can mean fine-grain control over permissions by content type, that level of role and permission definition is a bit excessive for simpler site structures.</p>
<p>To avoid complexity, I usually start with a trio of roles that are best assigned exclusively:</p>
<ul>
<li><strong>Editor (or author)</strong> - can create and edit content, but they cannot move that content to a published state</li>
<li><strong>Publisher</strong> - can do everything an editor can do and can also move content to a published/unpublished state</li>
<li><strong>Administrator</strong> - can do everything an editor or publisher can do, and can administer the site</li>
</ul>
<p>That’s pretty much all many sites need. With this model, you are trusting your trained editors (you are training them, right?) with the ability to create and edit as they need, but verifying the content is ready for publication with an additional review by a publisher.</p>
<p>With the optional <em>marked for deletion</em> state, you are also trusting your publishers to unpublish content that should be deleted with the intent that administrators will double check their choices.</p>
<h3 id="making-trust-more-and-verify-work-with-groups">Making trust more and verify work with groups</h3>
<p>A key aspect of large, enterprise websites is the notion of grouping content by a sub-organization. Groups provide hierarchy and information architecture to the site that often is used for the path alias (URL) of the content, the breadcrumb navigation, and menu navigation. I often call this sub-organization a “group” for the purpose of content modeling, but you could name it “organization” or “site” if preferred.</p>
<p>Groups could be a content type or it could be a subtype of content denoted through a field on a more basic content type. (For example, <em>page</em> of the subtype of <em>group</em>, <em>department</em>, etc.)</p>
<p>For the purpose of demonstration, let’s say you have a content type called “group” that represents the departments, divisions, and sections for a county (local governmental organization in the United States).</p>
<p>Editors and publishers for the site are assigned to groups for the purpose of determining which groups appear on their personalized dashboard. Content created within the site is assigned to groups via a “belongs to” or “displayed in” field.</p>
<h3 id="to-achieve-group-content-ownership-and-user-group-membership-you-need-the-following">To achieve group content ownership and user group membership, you need the following:</h3>
<ul>
<li>Content type called group</li>
<li>Content that belongs to groups (for example pages or news) have a field for Display in group</li>
<li>Users have a field called Groups to which you can assign groups within the organization to the user</li>
<li>Use views to create lists of my groups and my content with a contextual argument of the logged in user and with relationships that join the groups and its content to show a list of the content the user should be responsible for editing</li>
</ul>
<p>While the permissions for these users will still be sitewide, the list views serve the purpose of showing the content the user should edit—rather than all content they could edit.</p>
<p>A bonus side effect of this fairly open approach is that editors and publishers can rely on other users of the system to provide coverage for events like personal leave without an administrator having to explicitly get involved.</p>
<p>Paired with a <em>reviewers</em> entity reference field on content that should be reviewed—and perhaps some custom or contributed code to trigger a reviewer email message as needed—a trust more and verify approach can be quite robust and effective.</p>
<h2 id="the-group-module">The “Group” module</h2>
<p>The somewhat ambiguously named <a href="https://www.drupal.org/project/group">Group</a> module represents the more complex end of the access control spectrum.</p>
<p>Group is undergoing an extensive rewrite between version 8.x-1.x (version 1 for Drupal 8/9) and versions 2 and 3. Versions 2 and 3 are essentially the same, but version 2 is the upgrade path for sites that are on version 8.x-1x. If you are just getting started with Group version 3 in Drupal 9 or 10, start with version 3.</p>
<p>For the purpose of this post, Group is far too complex to describe everything it can do or how to fully configure Group for your content model, but hopefully the described scenario below may help you understand one possible use case for large sites divided into sub-organizations (groups) for access control.</p>
<p>First, you need to consider how many unique group types you need. For a public-facing website, you likely can get by with a single group type, but you may find yourself needing multiple access control models for intranets and extranets that have both open and restricted group types.</p>
<p>Once you have your group types defined, you can set roles per group type in addition to the sitewide roles provided by Drupal. Group also has the ability to set permissions to groups according to sitewide roles. This can be helpful for setting the ability for a sitewide administrator role to see all created groups.</p>
<p>After enabling the Group Node submodule, you can enable content plugins for the group and set permissions for each of your roles for what content that group role can create, edit, and delete.</p>
<p>Group defines all of the association between content and a group or a user and a group through what is called a Group relation entity. This entity serves as the glue between the different entity types—and is even fieldable for complex relationships. There is a small trade off with this complexity in that every group-based view needs group relationships (joins in the database query) for the results to be rendered. This can be very confusing to someone new to the group ecosystem.</p>
<p>While the group relationship entity and the permissions defined by the content plugin covers the basics of content access, creation, updates, and deletion, many organizations will need a bit more robust editorial workflow provided by the Content Moderation module from Drupal core. Unfortunately, or maybe fortunately given the complexity it would represent, Group is not aware of moderation states in workflows. To handle this, I typically have a sitewide role for granting publisher permissions and roles for group admin and editor within the group roles.</p>
<p>Having two layers of permissioning is complex, but it is the best option I’ve found to date for providing a high level of control to site administrators that need to assign strict permissions to users on their site.</p>
<p>A benefit of group-specific permissions to delete content, is that unlike the trust more and verify approach, you can assign the ability to delete content to a more trusted group administrative role rather than centralizing that capability at a sitewide administrator role.</p>
<p>Group can be extended by the Group Media and Subgroup modules. Group Media provides group relationships for media entities—allowing media to be “owned” by a group with permissions for creating and editing the media within a group. The Subgroups module allows permissions to be inherited up and down a hierarchy of groups.</p>
<h2 id="workbench">Workbench</h2>
<p><a href="https://www.drupal.org/project/workbench">Workbench</a> is a suite of modules that make it easier for an editor to edit content they created. Much of the Workbench experience is essentially pre-made views that can be overridden with additional customizations.</p>
<p><a href="https://www.drupal.org/project/workbench_moderation">Workbench Moderation</a> is a module that in many ways led to much of the functionality in the Content Moderation modules in core—and as such Workbench Moderation is deprecated as of Drupal 9 and not recommended for new sites on Drupal 9 or 10.</p>
<p><a href="https://www.drupal.org/project/workbench_access">Workbench Access</a> provides another layer of permissions that can be applied to sitewide roles through an access schema based on either menu or taxonomy hierarchies. And it’s fairly easy to retrofit a site with the “trust but verify” approach mentioned above.</p>
<p>Choosing between a menu and taxonomy access schema has some important considerations. Both menu and taxonomy have hierarchy (parent to child relationships). Hierarchy allows the site to be divided into sections and for users to be assigned to a section where their sitewide roles and permissions are applied. Per the module maintainers, “Note that the module only controls access to content editing. It does not provide any content filtering of access restrictions for users trying to view that content.”</p>
<p>Taxonomy hierarchy is limited to within a specific vocabulary. Taxonomy terms can be a rendered entity at a particular page path or a view that takes the contextual argument of a term ID. With a little configuration, term view pages or term pages with block views receiving a contextual argument could make a decent “landing page” for a group. However, taxonomy terms are missing some of the base fields that content types have such as author and revisions that are displayed in a revisions administrative view. (Under the hood, terms have had revisions since Drupal 10.1, <a href="https://www.drupal.org/project/drupal/issues/2936995">but at the time of this writing, the UI for taxonomy terms has not made it into core</a>.)</p>
<p>Menu hierarchy is the relationship between menu items. Menu item entities can point to any other Drupal entity with a rendered page path (for example, nodes, taxonomy terms, user profiles, custom entities) you can also create a menu item entity for views with a page display. Menu hierarchy is a little more flexible than taxonomy hierarchy, but the Menu UI (either the administrative page or per node) is a bit cumbersome with very large menus. I’ve worked around this in the past with a combination of the <a href="https://www.drupal.org/project/bigmenu">Big Menu</a> and <a href="https://www.drupal.org/project/cshs">Client Side Hierarchical Select</a> modules.</p>
<h2 id="taxonomy-access-control-lite">Taxonomy Access Control Lite</h2>
<p><a href="https://www.drupal.org/project/tac_lite">Taxonomy Access Control Lite</a> is the successor to Taxonomy Access Control (which was only maintained until Drupal 7). “This node_access module governs access to nodes based on the taxonomy terms applied to the nodes. A simple scheme based on taxonomy, roles and users controls which content is visible.” It adds no new tables, but has similar limitations to Workbench Access’s taxonomy access schema without the hierarchical inheritance within the concept of a “section”. For simpler site structures, this may be sufficient to grant needed access levels.</p>
<h2 id="what-about-organic-groups-og">What about Organic Groups (OG)?</h2>
<p>In Drupal 6 and 7, Organic Groups was my go to solution for group-based access control. It had a much simpler permission structure than the Group module—but it was still quite robust. Organic Groups allowed content types to be marked as either a group or a type that could be owned by groups.</p>
<p>The advantage of a group being a typical content type is evident when you are creating views that need to show groups alongside other content. It also means that your group content types have all the workflow and revision capabilities of all other content types. (Early in the development of the Group module, the group entity type could not be revisioned or assigned to a workflow. This is no longer the case, but it was a huge drawback for the first four years I found myself using Group.) OG also had a simpler relationship model, an entity reference field for both content and users.</p>
<p>Organic Groups was the backbone for Drupal Groups (<a href="https://groups.drupal.org">https://groups.drupal.org</a>) as well as Acquia’s Commons distribution that was popular throughout the Drupal 6 and 7 lifecycle. It could be used for creating just about any permission structure that needed membership and roles to define content access and editing.</p>
<p>With so much going for it, Organic Groups did not make a clean jump to Drupal 8. The development of Organic Groups was moved off of Drupal.org prior to the release of a Drupal 8 version. The rewrite that was undertaken for the Drupal 8 version of Organic Groups involved a huge amount of code deletion and slow progress in restoring the functionality that made it popular for D7 and D8 sites. Despite work on the D8 version as far back as 2014, there is still no stable release on Drupal.org. That’s not to say it is completely unused. Development has trickled along during that time, and those that are ready to take on a fair amount of custom code, can essentially write their own administrative interface for OG, but it takes very deep knowledge of OG to maintain it for a customer.</p>
<p>I no longer recommend Organic Groups for new projects because I don’t feel I could hand off the work after launch to make the product fully operationalized.</p>
<h2 id="what-solution-is-best-for-your-use-case">What solution is best for your use case?</h2>
<p>Choosing how complex to make your access controls on a Drupal website is largely about the needs for governance and distribution of permissions to users that can create and edit content.</p>
<p><strong>If your site is small</strong>, say a blog or a marketing site managed by a small group of editors, you can probably just <strong>configure core roles and permissions</strong>.</p>
<p><strong>If you have many editors for different segments of your organization</strong>, perhaps in a small government or educational website, use either a <strong>trust but verify content moderation workflow</strong> or a <strong>simple access control scheme with Workbench Access or Taxonomy Access Control Lite</strong>.</p>
<p><strong>If you have a very large number of editors or need strict access control policies to both create/edit and to access content</strong>, such as in the intranet of a large company or a social media solution for developer portal, then the best solution is likely <strong>the Group module with group types and group-specific roles and permissions</strong>.</p>
<p>Before you settle on the solution for your site, take the time to think through the value of added functionality versus the added complexity and maintenance. Larger more complex sites with more Drupal modules and configuration cost more to maintain, but that cost can be worth the improved features and functionality in many larger organizations.</p>
10 Translation Best Practices for Government Websites2023-02-20T00:00:00+00:00http://joshuami.com/2023/02/20/ten-translation-best-practices-for-government-websites<p>The following applies to any government website, regardless of country or level of government organization (nation/federal, state/province, local jurisdictions, etc.). That said, the more immigration a country has, versus emigration, the greater the impact created by the translation of governmental resources.</p>
<h2 id="1-know-who-will-benefit-from-translation">1: Know who will benefit from translation</h2>
<p>Governments should know demographic information and literacy rates through a combination of census, survey, and immigration data. Demographic data will explain the scope of the possible impact for those that are bilingual, but only by understanding literacy rates can an organization understand the possible reach of translated materials.</p>
<p>Understanding the population impacted by limited proficiency in the source language can be combined with knowledge of the non-government and government organizations supporting these populations.</p>
<h2 id="2-avoid-the-client-side-translation-crutch">2: Avoid the client-side translation crutch</h2>
<p>Many government websites rely on the deprecated <a href="https://developers.google.com/search/blog/2020/05/google-translates-website-translator">Google Translate Website Translator</a> to provide access to community members with limited proficiency in the provided language.</p>
<p>The Google Translate Website Translator provides essentially the same level of site translation as submitting a URL to <a href="https://translate.google.com/">translate.google.com</a>, but without the visual cues to help a community member understand that the translation is not being provided by the site.</p>
<p>Even the developers behind this service will admit it is a crutch to shore up the lack of full translation. The pandemic and the demand to translate health and safety information are why the <a href="https://developers.google.com/search/blog/2020/05/google-translates-website-translator">widget was partially revived from deprecation</a>.</p>
<p>Client-side (browser-based) widgets are an incomplete and misleading solution.</p>
<p>Policy makers, that speak the language of the government providing the content, assume the widget is providing access to critical content. Aside from providing an easy link to multilingual visitors, client-side translation is not increasing access.</p>
<p>Community members with limited language proficiency that use government websites tend to be familiar with services to help them translate their content. Browsers, such as Chrome or Edge, will automatically attempt to translate pages in a language other than the system or browser setting. There are also dedicated mobile apps to help with client-side translation.</p>
<p>If you speak enough of a language to find a page on a website, you probably know enough to figure out how to translate the page you found.</p>
<p>The real equity issue is finding content in the first place if you don’t speak the language.</p>
<p>Websites that do not have translated content using an alternate URL are not being fully indexed in target languages by search engines such as Google, Bing, DuckDuckGo, etc. That means people who speak a language other than those used in fully-translated content are unable to use their language to search</p>
<h2 id="3-review-translations-with-human-professionals">3: Review translations with human professionals</h2>
<p>At a minimum, governments should consider automating the translation of content using machine translation—usually via a web service that returns translated content—that is stored in the database for presentation at a unique URL. It’s important that this content is rendered as HTML as a language alternative for the source language rather than being rendered by Javascript.</p>
<p>While there are techniques for getting Google to index rendered Javascript, these approaches are not always reliable. Even though machine learning has progressed to the point that proxied translation is much more accurate, it still needs human review as of 2023.</p>
<p>Best practice for all government websites should be full translation at a unique URL in a culturally appropriate design that may be translated by an on-demand service (for speed) but must be reviewed by skilled human translators for accuracy.</p>
<h2 id="4-help-search-engines-rank-content-with-proper-metatags">4: Help search engines rank content with proper metatags</h2>
<p>Search engines need data to help determine the relative importance of a page—often in the form of metatags and HTML attributes. Search engines, such as Google, can use link elements (metatags) with a relationship (rel=”alternate”) and hreflang (hreflang=”es”) attribute to help rank content. In this approach, the search engine gives a bit more relevance to translations (alternates) of pages that rank highly in English.</p>
<p>Hreflang can be used to boost the search engine relevance of content both by language and by country or region. Most governments will find themselves using a single country code—or omit the country code—for all language codes used for their content.</p>
<p>Language codes are set by [look this up] a governing body that defines language and region codes. It’s not perfect—and it is always changing—but it serves as a means to ensure that the work you did to get your source content ranking will pass through to the content for your target languages.</p>
<h2 id="5-index-content-by-language-to-improve-site-search">5: Index content by language to improve site search</h2>
<p>In addition to making content available to search engines, full translation provides the opportunity to index text in a target language so that it can be found in site search. Sites that only use client-side machine translation lose the ability to submit forms to return search results in the target language. This is a huge loss of functionality! Would you find it acceptable if search could only return the occasional cognate or misspelling?</p>
<p>Languages have unique grammar and syntax. Languages have their own writing system (alphabet) and unique characters and accents. If the search engine cannot provide at least a rudimentary search in the language, it is likely to cause confusion and frustration.</p>
<p>For those that would argue that client-side translation is “enough” or “the best we can do”, consider the basic usability of a site with thousands of pages of content where search is incomprehensible.</p>
<h2 id="6-avoid-mixed-translation">6: Avoid mixed translation</h2>
<p>While mixed content (partially translated content presented on the same page as the source language) is indexed, partial translation can impact the usability for both the reader with limited language proficiency as well as native speakers from the source languages of the content.</p>
<p>Clear, concise content is difficult to achieve on a page with multiple paragraphs or sections of paragraphs in two or more languages—and the problem exacerbates as the number of languages presented increases.</p>
<h2 id="7-help-users-understand-the-languages-available">7: Help users understand the languages available</h2>
<p>Many websites that provide translated content will have a language switcher or toggle that allows users to quickly view the translated alternate for a page. Best practice is for this toggle to present the language label in the target language. So a site with translations in English, Spanish, Vietnamese, Russian, and Chinese would have language labels similar to this example from the Multnomah County Library.</p>
<p><img src="/public/images/multcolib-translation-navbar.png" alt="On multcolib.org, a translation bar appears on every page of the site with links to the page in alternate languages." /></p>
<p>When a language switch is presented as part of the navigation toolbar on a website, it is generally expected that it will switch into a fully translated version of that site with a starting point of the page being viewed.</p>
<p>For sites that do not have enough content translated to warrant a sitewide translation navigation, an in-page approach may be more appropriate. Here is an example from Portland.gov.</p>
<p><img src="/public/images/portlandgov-translation-menus.png" alt="Portland.gov uses two language menus. At the top of each translated page, the available language labels are used. In the secondary region, the list of translated titles is used for better language usability. " /></p>
<p>Finally, showing the related titles in the target language can be an excellent indicator to those with limited proficiency that are scanning the page for information. A title will actually tell them something about the content being linked—far more than a language label.</p>
<p>Combining all three techniques is overwhelming, but sites should strive to combine a language label menu in either the sitewide or in-page navigation near the top of the site design with related translated titles after the content.</p>
<h2 id="8-label-downloadable-files-in-other-languages">8: Label downloadable files in other languages</h2>
<p>When possible, information should be provided as content on a website (semantically-structured HTML) rather than downloadable files. Regardless of format, HTML is likely the easiest to structure for accessibility, HTML content can be structured and styled for a range of device sizes and for screen reading technology. Files can be semantically structured for screen readers, but often fail to deliver a usable experience on mobile devices. (What device types would you assume to be most available to the population needing translation?)</p>
<p>If files are preferred because of the way the information should be conveyed, such as flyers that might be posted in locations where speakers of that language are likely to see them, it is important to provide context for the language used in the file. This is often referred to as dual labeling.</p>
<p><img src="/public/images/washco-residential-collection-service-rates.png" alt="WashgingtonCountyOR.gov supports the showing of translated documents with a clear label including the title, file type, and file size." /></p>
<p>In the best examples of this approach, the source language of the document should be labeled next to the document name. Proper labeling is key to both the language speakers and those supporting those speakers that may not speak the target language.</p>
<p>Avoid links to files with text like “click here” or “download now” and explain what is to be downloaded with a meaningful title, the file type, and file size to be downloaded.</p>
<p>File size is important. Large file sizes are inequitable for government information as only those with enough data can download them. There may be file types that are unavailable to the device type being used to access the information. Additionally, some file types are more difficult to make accessible in any language.</p>
<h2 id="9-reduce-translation-costs-with-plain-language">9: Reduce translation costs with plain language</h2>
<p>The easiest way to reduce translation costs is to reduce the amount of text translated. Clear, concise content is easier for people to understand and costs less to maintain as well.</p>
<h2 id="10-strive-to-be-better">10: Strive to be better</h2>
<p>Delivering better government services to the most vulnerable populations is not always easy. The initial investment in your content management systems is incremental compared to the impact. That said, fully supporting all of the languages spoken in a community, including features such as permitting or online payment, may be prohibitively expensive for even the most well-funded jurisdictions.</p>
<p>By striving to be better, government organizations can improve service to their communities and improve equity.</p>
<h2 id="get-help">Get help</h2>
<p>Do any of these best practices sound like the next step your organization should take? If yes, you probably need some expert help to guide your roadmap.</p>
<p>Connect with me on LinkedIn (<a href="https://www.linkedin.com/in/joshuami/">linkedin.com/in/joshuami</a>) and let’s talk. I have worked with a wide range of vendors that specialize in translation and localization. I may be able to connect you to an organization that can boost your efforts to provide better content to your community in their language.</p>
Agile Development Scrum Terms and Recommendations2021-04-22T00:00:00+00:00http://joshuami.com/2021/04/22/agile-development-scrum-terms-and-recommendations<p>When starting a new agile project or product with a new team, I often find the need to establish common language. On simple projects, I usually walk the team through the following terms.</p>
<p>The toolset associated with these terms is usually some combination of Jira with Github/Gitlab/Bitbucket and a CI/CD toolchain that connects to hosting. There are many possible toolsets, but the agile product management process that uses scrum methodology nearly always has the same dictionary of terms.</p>
<p>To facilitate some larger projects, and to try and get down definitions that meet my own needs for large Drupal CMS implementations, I am writing this handy guide to the most used terms that I use in consulting.</p>
<p>Rather than present an alphabetical list, I find it best to start with some of the largest concepts and work them down and then back up again in complexity.</p>
<p>So… where to start?</p>
<h2 id="product-roadmap">Product Roadmap</h2>
<p>You are not working solely on a “project” anymore—even if you may call it a project because it has a planned end date and some milestones to get to that state of readiness—note that I didn’t use completeness; most products are never “complete” so long as they are still delivering value.</p>
<p>Once you take on agile methodology to manage your applications, software, or process, you aren’t really working on a strict schedule defined by predecessors and successor tasks as is common in waterfall style project management. That doesn’t mean you can’t project a schedule, but no single task is going to get a start and end date and show up on a gantt chart.</p>
<p>Instead, you are going to break your work into logical pieces related to your product goals, we sometimes call this a roadmap. You are drawing this map as you go. It will change.</p>
<h2 id="themes-and-epics">Themes and Epics</h2>
<p>Your first level of defining your product roadmap is quite broad. If the product is large enough, you may find that you need to group work that represents deliverable features (<strong>epics</strong>) into larger categories of work based on your goals (<strong>themes</strong>). Not every product roadmap requires themes, but every product roadmap will have epics to help with planning.</p>
<p>If you use themes, you’ll want to keep them broadly-related to the goals of your project. Good examples of themes in large Drupal CMS implementations would include accessibility, mobility, maintenance and devops, and localization (or internationalization if global). While you might use these one-word labels to describe your themes, you should spend some time detailing what success looks like for each.</p>
<p>Epics are the larger features that are related to a theme. Epics themselves are not a deliverable, but they help group the work to achieve a large change to the site. Epics might get target schedules applied to them so that you can show these schedules on a roadmap that looks a little like a gantt chart but is usually more flexible.</p>
<h2 id="stories-tasks-and-bugs">Stories, Tasks, and Bugs</h2>
<p>While epics help you group your work to plan, the deliverable work for your team will come in the form of stories, tasks, and bugs. (Some folks call bugs “defects”, but it seems so much more satisfying to squash a virtual bug—and much less karmically intense than squashing a live bug.)</p>
<h3 id="stories">Stories</h3>
<p>A <strong>story</strong> is a discrete set of work that delivers value to your product. Some teams will only use a story to define work that will be delivered as a code change. I am not that picky. To me, a story is anything of value. In addition to code, value could be delivered in the form of documentation, research, analysis, design, or content creation. You will need to set the norms with your team.</p>
<p>Each story should fit into the following criteria that are sometimes abbreviated with the mnemonic I.N.V.E.S.T:
Independent: self-contained in a way that allows the work to be released without depending on another story. (Though you can choose to build upon previous stories that were delivered before you release a larger feature.)</p>
<ul>
<li><strong>Negotiable</strong>: leave room for conversation as the team is working on the story. Don’t write a contract so much as ensure that you have enough guidance to make the work achievable.</li>
<li><strong>Valuable</strong>: it’s easy to allow your backlog to grow with stories that don’t represent a lot of value to the end user. Make sure you clean up your backlog and only put work into your work plan (more on that soon) that truly improves the product.</li>
<li><strong>Estimable</strong>: you need to be able to put a solid guess for how much effort and time a story will take when you start work on it. There are a couple of exceptions to this I will describe below.</li>
<li><strong>Small</strong>: I would actually argue that this is more of an “as small as feasible” requirement. You might find that very small stories make your deployment pipeline slow to a crawl, but large stories can have a similar impact.</li>
<li><strong>Testable</strong>: some folks swear by test driven development (TDD). I find that is a bit of overkill for most Drupal CMS projects, but you should know what you intend to test via automated tests and what you need to verify through a quality assurance test from someone on your team. Write this up as acceptance criteria on every story in your sprint. You can modify this test through negotiation—see above—but you need to know that the final thing you deliver is adding the value you hoped to add.</li>
</ul>
<p>(I’m not a huge fan of mnemonics, but sometimes they are helpful.)</p>
<p>I have a couple of exceptions to this recommendation to make it possible to include work in your roadmap that is hard to estimate.</p>
<h3 id="pointing-stories">Pointing Stories</h3>
<p>What is the value of your story? I’ve had teams in the past that joke “points mean prizes”. While I have never actually awarded prizes for “most points completed”, I really appreciate team members that figure out the ability to estimate how many stories they can take in a given work plan. (I’ll describe sprints below; sprints are iterations of work.)</p>
<p><strong>Points</strong> are not strictly a time estimate. Ideally, points represent the difficulty of a story and the general size. There are many different pointing approaches that each have their pros and cons.</p>
<h4 id="t-shirt-sizing">T-Shirt Sizing</h4>
<p>Small, medium, large, and extra large can form the basic of defining story complexity. These labels are handy to abbreviate following the standards used by the clothing industry.</p>
<h4 id="1-to-5-or-1-to-4">1 to 5 (or 1 to 4)</h4>
<p>Is it a 1, 2, 3, 4, or 5? This is probably my least favorite. You have to have an extremely high functioning team to agree what each of these numbers really mean. (It would almost be better to not point at all.)</p>
<h4 id="fibonacci-sequence">Fibonacci Sequence</h4>
<p>The best-named pointing scheme by far! The Fibonacci sequence creates a list of numbers based on adding the previous number to the next number: 1, 2, 3, 5, 8, 13, 21. (1+2=3, 2+3=5, 3+5=8, etc.) This is mathematically beautiful pointing! (The nautilus shell grows based on the Fibonacci ratio—so this pointing scheme happens in nature.)</p>
<h4 id="doubling-closest-to-time-based-pointing">Doubling (Closest to Time-based Pointing)</h4>
<p>My personal favorite for teams new to pointing stories is a doubling scheme: 1, 2, 4, 8, 16, 32.</p>
<p>Doubling is the closest to time-based pointing. Some would say this is a reason not to use this approach, but everybody can get the idea that a point is roughly an hour. By enforcing doubling, you basically tell your team that hour estimates are not that accurate. You are more accurate with smaller stories and less accurate the more complex they get. 16 doesn’t mean 16 hours, it means the story is probably a couple days of work—give or take. 8 is not a commitment to finishing a story in a day, but it does mean you think the story will take you “about a day”.</p>
<p>My teams limit the use of 32 point stories to <em>spikes</em>. What’s a spike? A spike is a story that is really hard to estimate because it involves investigation and analysis. (The lore behind the name spike suggests it was related to mountain climbing and the act of setting a spike to help define your path and anchor you to the parts of the route you already know. That lore might be true—I hope it is.)</p>
<p>You can point spikes that are smaller than 32 points—with the expectation that they are “time-boxed” to less time.</p>
<p>Some teams choose not to point spikes, but personally, I feel that makes it hard to balance your team members’ load of work in a sprint.</p>
<p>Pointed stories are the core of planning your product roadmap, but there are other types of work that will get added to your backlog and into your sprints.</p>
<h3 id="tasks">Tasks</h3>
<p>Tasks are quick work that can be completed without a lot of complexity—and usually not much direct value to the product. Each team may choose to use tasks differently. Most teams choose to not point tasks.</p>
<h3 id="bugs">Bugs</h3>
<p>Dangit! There’s a bug!</p>
<p>Seriously, bugs happen. When you find a defect in your product that needs to be fixed. Add it to your backlog and prioritize it. Most teams don’t point bugs—similar to spikes, they are extremely difficult to estimate.</p>
<h2 id="sprints">Sprints</h2>
<p>So now, you have your themes > epics > stories/tasks/bugs (and your stories are pointed) and you are ready to plan a <strong>sprint</strong>.</p>
<p>I understand why they are called “sprints”—they are a short period of time; they represent a short term goal—though I wish the term was “iteration” or “period”. Constantly “sprinting” is kinda exhausting. Building a great software product is really more of a long-distance sort of run.</p>
<p>That said, sprints are the frequency of your planning and agile rituals. You don’t want to plan for sprints that are too long (a month-long sprint is really the maximum) or too short (less than a week would be kinda insane). I’m a fan of 2 or 3 week sprints, but it really does depend on the team.</p>
<p>Sprints have three types of regular meetings (rituals) that define their cadence: sprint planning, standups, grooming meetings, and sprint retrospectives.</p>
<h3 id="sprint-planning">Sprint Planning</h3>
<p>At the start of each sprint, you want to bring the team together to determine the following:</p>
<ul>
<li>What is the start and end date of the sprint?</li>
<li>Who is available for the sprint—and how much? (Plan for time off!)</li>
<li>What are the goals of the sprint?
What stories, tasks, and bugs should be included from the backlog?</li>
<li>Who gets what stories as their assignments?
At the end of sprint planning, you should have what amounts to a team contract regarding what you’d like to complete in that period of time.</li>
</ul>
<h3 id="standups">Standups</h3>
<p>The name <strong>standup</strong> comes from the idea that you shouldn’t sit down for this quick meeting that should ideally last less than 15 minutes. You want to hold a standup everyday that you aren’t planning or holding a retrospective—we don’t want to meet so much that we don’t have time to get things done.</p>
<p>In a standup, you are just keeping each other up to date on your progress towards the sprint plan. There are three questions you may answer in your brief daily update:</p>
<ul>
<li>What I did yesterday…</li>
<li>What do I plan to do today…</li>
<li>What is blocking my work…</li>
</ul>
<p>I have yet to work with a team that is truly great at the timing of standups. My best attempt at getting a team to stick to the timing involved a 2 minute timer on my phone that would quack if the team member went over time. At least we got to laugh at the time-keeping device.</p>
<p>The most important part of standups is learning the blockers and figuring out a way around those blockers.</p>
<h3 id="grooming">Grooming</h3>
<p>Another weird scrum word, <strong>grooming</strong> is the act of making sure your backlog is ready for the next planning session. Every sprint should have some time to work through negotiating the details of your stories to make sure you can point them and put them into an upcoming sprint.</p>
<p>I recommend about an hour per week of sprint dedicated to grooming. If your team velocity allows, you could easily spend up to 3 hours grooming stories for future sprints.</p>
<h3 id="sprint-retrospectives">Sprint Retrospectives</h3>
<p>At the end of each sprint, the team should conduct a <strong>retrospective</strong> (AKA “retros”) and to talk through the following:</p>
<ul>
<li>What went well this sprint? (Celebrations)</li>
<li>What could have gone better? (Blockers to the plan)</li>
<li>What are we going to change going forward?</li>
</ul>
<p>High-functioning teams using scrum for product development will use this time for honest and reflective feedback.</p>
<h3 id="demonstrations-with-stakeholders">Demonstrations with Stakeholders</h3>
<p>Often tacked on to the end of the retrospective meetings, demos are the key to making sure you are doing the correct work and give your stakeholders a chance to add feedback. Stakeholder feedback should be captured as future epics, stories, tasks, and bugs.</p>
<p>Demonstrations can be a mix of show and tell and usability testing with stakeholders actually trying out the feature while the team watches and takes notes.</p>
<h2 id="optional-but-helpful-rituals">Optional (But Helpful) Rituals</h2>
<h3 id="planning-retreat">Planning Retreat</h3>
<p>You may find that you need to spend an extended time planning the roadmap with the team on a quarterly basis to build up a longer term roadmap and weed out stories that may never be addressed. Think of these sessions as extended and more intense grooming. You might even meet for 3-4 hours several times over the course of a week to talk through product vision and goals or two break down collected analytics and study of usability test artifacts.</p>
<p>Don’t plan a sprint over the top of these retreats. Use the time as a way to reset and get back to a shared vision. This is especially helpful for larger products that have multiple scrum teams that collaborate on separate components. Getting these teams together to hear presentations from all the teams involved is invaluable in creating a cohesive product.</p>
<h3 id="pair-programming">Pair Programming</h3>
<p>During any given sprint, you may choose to assign a story to more than a single developer to ensure knowledge sharing/skills or to just tackle a particularly difficult problem. Giving team members the freedom to work together rather than in isolation is a great way to foster creativity and a better final product.</p>
<h2 id="when-is-the-scrum-project-complete">When is the Scrum Project Complete?</h2>
<p>Software applications and website platforms tend to go through a software lifecycle that starts with investing in the product then transitioning to more of a sustain and maintain and eventually to deprecate and/or replace.</p>
<p>Scrum is appropriate for organizations investing in their application.</p>
Launching Portland.gov2020-06-30T00:00:00+00:00http://joshuami.com/2020/06/30/launching-portlandgov<p>On June 16th at 7:30 a.m., we switched the homepage of PortlandOregon.gov over to <a href="https://www.portland.gov/">Portland.gov</a>. The switch officially moved a platform two years in the making out of beta.</p>
<h2 id="easy-to-find-access-and-understand">Easy to find, access, and understand</h2>
<p>When I started on this project in January of 2018, I knew this was a huge project. Two years later, the vision for what we wanted to achieve still holds up:</p>
<blockquote>
<p>Portland.gov is a community-focused website where services and information are easy to find, easy to access, and easy to understand.</p>
</blockquote>
<p>We took time early in the project to test whether we were hitting these goals through actual user testing. Before the pandemic, this meant moderated tests in some of our public service locations like the development services center, the Revenue Division’s payments desk, and at the Water Bureau’s customer service center.</p>
<p>The chosen platform is Drupal, an open source content management framework used in ambitious web solutions by companies, governments, and educational institutions around the world.</p>
<p>Drupal’s flexibility to create a tailored content model allowed us to focus on the vision.</p>
<h3 id="easy-to-find">Easy to find</h3>
<p>We have put a ton of work into search. After choosing Solr as our search engine, we spent quite a bit of time tweaking our index and search pages. We wanted to make sure services and programs that are most relevant will be at the top of our results.</p>
<p>The content model also supports breaking out key information by type so that you can search events, news, advisory groups, or strategic projects. By using facets, the process is similar to a commerce search. Instead of picking a television by brand, size, and features, people will be able to narrow down news from the last year focused on garbage and recycling, or perhaps events by whether it was a public meeting open for testimony.</p>
<h3 id="easy-to-access">Easy to access</h3>
<p>Accessibility has been at the front of our work since the start of the project. People should be able to access their government and services regardless of ability or the type of device the need to use.</p>
<p>Mobile-first design is an equity issue. Large file sizes cost community members their data plan to access content. As much as possible, we should keep the content simple and lightweight so that people are not prevented from access.</p>
<p>While not perfect, we have been able to steer content editors in the bureaus to use less PDFs and Word documents and instead focus on creating web pages. Our editor experience essentially forces content creators to make better decisions about their content structure. Content is semantic and easily consumed by screen reader or via keyboard navigation.</p>
<h3 id="easy-to-understand">Easy to understand</h3>
<p>It has been a huge effort to retrain how people write a government service. There is a tendency in the government to write bureaucratically. We are shifting that to clear, concise, and simple language where possible. Pointing our editors to tools like <a href="https://www.grammarly.com/">Grammarly</a> and <a href="http://www.hemingwayapp.com/">Hemingwayapp</a> have let them see just how complex their language has become. These tools give content creators feedback to improve their writing.</p>
<p>We are also planning some pretty impressive translation capabilities later this year. Language alternate content will be available in the City’s 10 safe-harbor languages as actual web pages rather than attached PDFs.</p>
<h2 id="shifting-from-a-project-mindset-to-a-product-mindset">Shifting from a project mindset to a product mindset</h2>
<p>With such lofty (but achievable) goals, the migration to the new platform required more than just a migration project mindset. We were really rebuilding both the platform and the content strategy from scratch.</p>
<p>To facilitate that rebuild, we had to rethink the way we planned work and how we trained content editors.</p>
<h3 id="planning-and-execution">Planning and execution</h3>
<p>The toolset to get work from an idea to reality needed to be the same sort of tools that a start up could use to bootstrap a team to build a product. We use a lot of the tools you would expect, but are rare to find in government:</p>
<ul>
<li><a href="https://www.atlassian.com/software/jira"><strong>Jira</strong></a> for story writing, epic tracking, and agile sprint planning</li>
<li><a href="https://lando.dev/"><strong>Lando</strong></a> for local development run in Docker containers</li>
<li><a href="https://github.com/"><strong>Github</strong></a> for a code repository and pull-request-based workflow</li>
<li><a href="https://circleci.com/"><strong>CircleCI</strong></a> to automatically build feature branch sites for automated testing and QA</li>
<li><a href="https://pantheon.io/"><strong>Pantheon</strong></a> as a Drupal-specific host with its own git workflow and dashboard integrations</li>
</ul>
<p>These all seem pretty straightforward for organizations that build products, but outside of some limited Github usage, none had been used by the City of Portland before.</p>
<p>Our sprint planning frequency and pace has changed a lot over the course of the project. Early on, we settled into a regular 3-week sprint pattern that worked pretty well. During the pandemic, we shifted to a more frequent, one-week sprint. All sprint lengths have advantages and disadvantages, but it was clear that remote work required more frequent planning and communication—and did not lend itself to the all-day planning session with markers and sticky notes that was a huge boon to our 3-week sprints.</p>
<h3 id="training-and-education">Training and education</h3>
<p>During the course of the migration and platform building project, shifting the way we train from a quarterly new-editor training to ongoing, twice-weekly office hours served two purposes.</p>
<p>First, it gave editors many more opportunities to ask questions about their specific content and the user journeys they were creating. We have tried to have editors shift from just creating random content to creating content with intention that puts thought into where a community member might first start their quest for information:</p>
<ul>
<li>noticed a billboard or flyer</li>
<li>followed a featured post in social media</li>
<li>directed to the information by a trusted advisor</li>
<li>received an email from a transactional business system</li>
<li>subscribed to updates from the bureau or program</li>
<li>searched for the information on Google the search engine of their choice (I’m excited to see the growth in DuckDuckGo!)</li>
<li>searched on Portland.gov because they assumed it was a city provided service</li>
</ul>
<p>Each of these paths have important considerations for how the content is created—especially for search—and for how the content is later shared out. Communication plans are important.</p>
<p>The second advantage of twice-weekly office hours was a chance to hear what features our editors needed to achieve their communication goals. These sessions have led to some of the best features of the new platform—both features for the editors and community members using the site.</p>
<h2 id="what-i-would-do-different">What I would do different</h2>
<p>While the migration and new platform have been a success, there were definitely a few growing pains. If I could go back and start this project again, I would have made a few changes to the structure of the project and the team.</p>
<p>If you are undertaking a platform of this scope, I would recommend the following team:</p>
<h3 id="product">Product</h3>
<p><strong>Product Manager</strong> leads the prioritization of the features of the platform and acceptance of the user stories at the end of each work sprint. The product manager, through their role of acting as the customer surrogate, is really the decider of what the product should achieve. They should have a clear way to test that the product is meeting the goals. Product managers should also lead the design process.</p>
<p>Content management systems, and the sites built on them, that look good but are hard to use are usually the result of a push for form over function. If you don’t meet the basic need of showing the content to the end user in a way they can understand, your platform is not a success. This is another reason to conduct real user testing—both for overall experience and accessibility.</p>
<p><strong>Content strategists</strong> will help your communications stakeholders understand their audience better. They can often provide critical rewrites that make content simpler as well as insights that lead to better features for editors and end users.</p>
<p><strong>Product designers</strong> are key. Ideally you want design support that has used Drupal before and understands the platforms strengths and limitations. (See my notes on themers below.) It is also important that the design expectations match the goals of the product and content strategy.</p>
<p><strong>Quality assurance</strong> can be a dedicated role or an aspect of another role. The best QA teams have tools that make their work something that can be automated. Functional testing and visual regression testing make the work of deploying change to a platform easier—and contribute to the stability as well.</p>
<h3 id="technology">Technology</h3>
<p>A <strong>technical lead</strong> should make sure that every architectural decision meets the organizational goals for security, performance, and stability of the application. A strong technical lead brings balance to the product and ensures that the parts of user experience tied to page load speed and time to first print are considered.</p>
<p><strong>Developers</strong> it goes without saying that you will always need to customize a content management system to meet the needs of the site. Great Drupal developers know when to write a small custom module versus when to bring in powerful contributed modules for the larger community that are well tested and configurable to the need. If you are replatforming and want to build up an existing team to learn a new technology, build learning time into your overall schedule and expectations.</p>
<p>You will have technical debt when you are two years into a new product development—regardless of team experience—but there will be more if the project is also the first learning experience. While this is an area I thought went well in the Portland.gov process, I also recognize a lot of our technical debt came from learning a new platform at a time when that platform was solidifying. Drupal 8 is a phenomenal open source platform, but I would argue that at the beginning of 2018, it was not yet ready for the ambitious product we were building.</p>
<p><strong>Themers</strong> are a bit unique to the world of CMS platforms. The best “devsigners” are a little bit artist and a lot developer with a deep knowledge of their chosen platform. When using a tool like Drupal, a good theme can make the difference between whether your team members can use core and contributed modules without having to ask a designer to first tell them what it should look like. The more you override in a theme, the harder it is to ensure that theme will work with new features picked up through contributed development and simple configuration.</p>
<h3 id="process">Process</h3>
<p>A dedicated <strong>scrum master</strong> can make the difference in whether a team feels harried or content with the workload. They are also critical for unblocking teammates by scheduling all the meetings/trainings/presentations that are needed to keep things moving. Great scrum masters do more than lead the rituals of an agile project; they are the utility player that helps keep everything efficient and moving forward.</p>
<h3 id="other-roles-you-may-need">Other roles you may need</h3>
<p>Depending on the size of the project and the toolchain supported, you may need a dedicated DevOps engineer. You may need a dedicated analytics specialist to find patterns in your data and help your content strategist make good decisions. There could be times you want to pull in a business analyst to help detail the requirements of an integration on the platform.</p>
<p>Perhaps the most important role to fill in projects of this size is that of content migrators that can manually do tasks that are too complex to automate or require significant rewriting. Under the direction of a good content strategist, a couple of fresh-from-college, or even currently in college, interns can go a long way towards getting a lot of content moved quickly. In fact, this project introduced me to one of the best content migrators I’ve ever seen. He’s a junior… in high school. (How’s that for overachieving?)</p>
<p>Another place to look for content migrators is within the organizations migrating. Look for folks with excellent written communication that want to try something new in their career.</p>
<p>Whether they are college interns or professionals with years of experience, they may find they like the work and pivot accordingly. Or they may let you know just how challenging large-scale websites can be and move on to other work with some great new skills and a deeper understanding of web technology. Either is a boon for both the migration and their careers.</p>
<h2 id="thoughts-after-the-homepage-launch">Thoughts after the homepage launch</h2>
<p>If you had asked me two and a half years ago whether I thought it would take this long to switchover the homepage, I would have assumed a much shorter timeline. During the course of this particular work, I learned just how impactful centralized leadership is to government projects of this scope. We had bureaus that were nearly completely migrated one year after our initial alpha release of the services directory in November of 2018. We have other bureaus that we had to figure out creative ways to redirect their content back to the old platform for the homepage switch to occur. These bureaus will probably take all of the remainder of 2020 to complete their migrations.</p>
<p>So what was the difference? In one case, a dedicated communications specialist leads their stakeholders through the migration with determination and efficiency by hiring an intern and participating thoroughly with the project team. In some of the lagging migrations, the biggest difference is a lack of dedicated communications personnel that have a focus on digital communications. For others, moving to a new platform was not a priority until it was clear the platform was going to be adopted.</p>
<p>For myself, this work with the City of Portland has been quite an experience. I have met some incredible people that are really passionate about delivering services to the community they serve.</p>
<p>My last few roles have steadily pushed me more and more into strategic management and leadership rather than actual production. This project required that I actually show the work rather than asking folks that have never used a platform to go out and learn from scratch what I know is theoretically possible. I have pushed myself professionally—in many cases learning new skills so I could turn around and teach. That opportunity to learn new skills was invigorating.</p>
<p>So what’s next? Most of the remaining migration work will be done by the bureaus, but later this summer we will be launching a new employee intranet as well as turning on translation management features on Portland.gov that will allow editors to create fully-tagged language alternate versions of their content. Also, there are over 1,550 webforms, and the related processes, that will be migrated into new customer service management tools. Finally, at some point, there will be an effort to theme, retire, or retool the 70 or so applications that were literally written into the Coldfusion platform that the custom CMS was built upon.</p>
<p>All this ongoing work furthers the adage that a website is a product, evergreen and part of a lifecycle, rather than a project.</p>
Chasing Core: Why the Drupal 8.7 Update Was So Difficult2019-06-28T00:00:00+00:00http://joshuami.com/2019/06/28/chasing-core-drupal-8-7-update-difficult<p>Drupal 8.7 was released nearly two months ago and it took me that long to upgrade a complex site. Well… I should say it took me about 40 hours of research and development time. I worked on other things during the numerous local rebuilds it took. I am not proud of how long it took, but I am pretty impressed that I figured it out. I think I tried about 10 different approaches before I landed on the magic combination.</p>
<p>So why is chasing Drupal Core to the 8.7 update so difficult? A normal update to core is as simple as <code class="language-plaintext highlighter-rouge">composer update</code> followed by <code class="language-plaintext highlighter-rouge">drush updb</code> and a cache clear (<code class="language-plaintext highlighter-rouge">drush cr</code>). The challenge of this upgrade comes down to three things.</p>
<h2 id="i-let-the-configuration-become-too-complex">I let the configuration become too complex</h2>
<p>It happens to all of us that work in Drupal long enough. We start a new team on a new project that is big and complex and we teach them to Drupal along the way. In the process, one developer adds a module for this and another adds a module for that.</p>
<p>We started with Acquia Lightning, but our approach has bloated along the way to include modules like Groups (Organic Groups wasn’t ready yet), Paragraphs (Layout Builder wasn’t ready yet), Display Suite (Layout Builder doesn’t really do this), Content Lock, migrations, Solr Search API (which we have to fool composer to load a more modern version than what Acquia uses), Webform, and more.</p>
<p>On their own, these are all amazing tools to have when building a complex site. But suddenly—as if it happened overnight, ha—our <code class="language-plaintext highlighter-rouge">composer.json</code> file is huge and there are patches galore.</p>
<h3 id="speaking-of-patches">Speaking of patches</h3>
<p>It might be a good rule of thumb to limit your total patches to no more than twice the years you have worked with Drupal 8. Patches are amazing as they allow you to add a fix that is coming in the future with very little effort. The problem is that a patch is a promise of how code will be added to files in the future. Changes accepted before the patch in that future release can make the patch no longer apply.</p>
<p>As a developer in Composer world, if you apply too many patches, you will spend a lot of time rerolling patches in the Drupal issue queue—or grabbing it from others when they beat you to it.</p>
<p>When possible, you should only use a patch that you know is committed to the next version so that you can safely remove that patch upon upgrade. Let’s face it. If you have a big project that needs the patch. You are going to patch away anyway.</p>
<h3 id="configuration-is-everywhere">Configuration is everywhere</h3>
<p>Every edit you make to an entity type (content node, taxonomy term, menu item, etc.) changes a configuration file. In the site we upgraded, I had 1,499 configuration files. (And that is not counting configuration splits for preproduction and production environments.)</p>
<p>Configuration is awesome when it helps you keep your database in sync, but it can be a nightmare when something is corrupted or a schema changes.</p>
<h3 id="composer-will-try-to-update-everything">Composer will try to update everything</h3>
<p>Dependency management is pretty amazing. When I built up the team at the Drupal Association to support Drupal.org, it became apparent pretty quickly that if we wanted Drupal 8 to succeed, we needed Composer integration to succeed.</p>
<p>Composer is amazing at what it does, but I would not say that it is easy to understand all the inner workings. You have to understand semantic versioning (semver) deeply to have success with Composer.</p>
<p>When everyone in your dependency tree is doing their job, and your <code class="language-plaintext highlighter-rouge">composer.json</code> is well structured, running <code class="language-plaintext highlighter-rouge">composer update</code> will give you all the new stuff you need from Drupal and all its dependencies as well as put your contributed dependencies into an easily understandable directory structure that sorts code from others and the code you have customized in your git repo.</p>
<p>I found that this particular Drupal update needed as few concurrent upgrades running as possible. That meant pinning Lightning to the 3.x release (`composer require acquia/lightning:^3.2), as well as updating a lot of underlying modules like Entity Views Attach, Field Defaults, Easy Breadcrumb, and getting core up to the latest release of 8.6.17.</p>
<p>If it hadn’t been for this gem of a <a href="https://www.previousnext.com.au/blog/patch-drupal-core-without-things-ending-up-corecore-or-coreb">blog post</a>, I don’t think everything would have updated cleanly to Drupal 8.6.17 because a couple of my patches were writing to <code class="language-plaintext highlighter-rouge">core/core</code> instead of just <code class="language-plaintext highlighter-rouge">core</code>.</p>
<p>I should also note that the final build required that I <a href="https://www.drupal.org/project/lightning/issues/3056074">pin our version of Lightning Layout per a known issue</a> as well as stick Search API Page (<code class="language-plaintext highlighter-rouge">composer require drupal/search_api_page:1.0.0-alpha12</code>) and the Groups module (<code class="language-plaintext highlighter-rouge">composer require drupal/group:1.0.0-rc2</code>). It is not that either of those modules are a hard dependency, but their upgrades were failing for other reasons and I wan’t to isolate the Drupal 8.7 upgrade as much as possible.</p>
<h2 id="core-decided-to-add-revisioning-to-taxonomy-terms-and-menu-links">Core decided to add revisioning to taxonomy terms and menu links</h2>
<p>I’m actually okay with the decision. If I had it my way, every entity in Drupal would just be an entity with all the same field and workflow options. Let me publish and unpublish all the things!</p>
<p>The approach was solid, but the ability to apply that approach is heavily dependent on a fairly simple Drupal installation. Tests can only uncover so much. As a result, there are a significant number of sites that have struggled to upgrade with issues like:</p>
<ul>
<li><a href="https://www.drupal.org/project/drupal/issues/3052204">Unable to update Drupal 8.6 to Drupal 8.7; Field storage definition for ‘type’ could not be found in file_update_8700()</a></li>
<li><a href="https://www.drupal.org/project/drupal/issues/3052318">update from 8.6.15 to 8.7 fails due to menu_link_content</a></li>
<li><a href="https://www.drupal.org/project/drupal/issues/3039586">Cannot rename tmp_2362aemenu_link_content_revision to menu_link_content_revision</a></li>
<li><a href="https://www.drupal.org/project/drupal/issues/3052464">Cannot update to 8.7.0 because of taxonomy_post_update_make_taxonomy_term_revisionable</a></li>
</ul>
<p>I have been following these issues, and more, over the past month to figure out what exactly was causing database updates to fail on my attempts to upgrade.</p>
<p>Solutions to the corrupted tables ranged from writing custom modules with update hooks to unset configuration that was in conflict to database queries to remove tables or add fields manually. I’m not a big fan of altering a Drupal database under anything other than extreme circumstances. The abstraction that makes Drupal so powerful for configuration and site building also makes it incredibly complex. Any database change made without absolute understanding of the complex joins that will be made for entities or views is fraught with danger.</p>
<p>So how did I manage to fix this mess without touching anything more than a composer.json and updating a couple of views. The short answer is that I had three patches in place that conflicted with the entity update code that was a part of the changes that added revisioning to taxonomy terms and menu items.</p>
<p>The first, <a href="https://www.drupal.org/project/drupal/issues/2953331">Add a views sort handler for sorting content by moderation state</a>, I decided we didn’t need. Sorting by moderation state is an administrative task of questionable value. You might group by moderation state or filter by moderation state, but do you really need to sort. So I removed that patch and updated the 5 administrative views with tables that were trying to sort by moderation state. These changes were all in the table settings for that view display.</p>
<p>The next two issues, <a href="https://www.drupal.org/project/drupal/issues/2797583">Dynamically provide action plugins for every moderation state change</a> and <a href="https://www.drupal.org/project/drupal/issues/2174633">View output is not used for entityreference options</a>, were code that we needed. However, Drupal 8.6 needed the code in a different place than in Drupal 8.7. (See the patches problem statement above.) Because this update required multiple builds that ran all the way from local to multidev to dev to test to live, I had to modify the composer.json file to have a different patch when updating Lighting 3.2.x than from updating Lighting 4.0.x.</p>
<h2 id="order-of-operations-in-automated-builds-is-very-important">Order of operations in automated builds is very important</h2>
<p>One of the developers on the team at the City of Portland took a zsh alias that I had set up and turned it into a really slick little bit of tooling in our <code class="language-plaintext highlighter-rouge">lando.yml</code>. I really pretty heavily on these Lando commands to make sure that we don’t forget a command when we are setting up a local environment for a successful build.</p>
<p>This upgrade was particularly challenging as it has a pretty critical list of updates that require a configuration export <strong>after</strong> the updates are complete. This made me reconsider my scripts… which I will now provide here for those that might want to incorporate this into their Drupal workflow.</p>
<p>I also use Oh My Zsh with some Git shorthand to reduce my typing. I’ll provide the full commands alongside.</p>
<h3 id="starting-a-new-branch-from-master">Starting a new branch from master</h3>
<ol>
<li><code class="language-plaintext highlighter-rouge">gco master</code> (<code class="language-plaintext highlighter-rouge">git checkout master</code>)</li>
<li><code class="language-plaintext highlighter-rouge">lando latest</code> (The following commands run from the project root.)
<ol>
<li><code class="language-plaintext highlighter-rouge">mkdir -p /app/artifacts</code></li>
<li><code class="language-plaintext highlighter-rouge">rm -f /app/artifacts/database.tar.gz</code></li>
<li><code class="language-plaintext highlighter-rouge">terminus backup:get portlandor.dev --element=database --to=/app/artifacts/database.tar.gz</code> (We use Pantheon for our hosting, but all the big Drupal hosts have similar commands you can run.)</li>
<li><code class="language-plaintext highlighter-rouge">database: cd /app && /helpers/sql-import.sh artifacts/database.tar.gz</code></li>
<li><code class="language-plaintext highlighter-rouge">drush cr -y</code></li>
</ol>
</li>
<li><code class="language-plaintext highlighter-rouge">lando refresh</code> (This command gives us a clean starting point with our database.)
<ol>
<li><code class="language-plaintext highlighter-rouge">composer install</code></li>
<li><code class="language-plaintext highlighter-rouge">drush cr -y</code></li>
<li><code class="language-plaintext highlighter-rouge">drush updb -y</code></li>
<li><code class="language-plaintext highlighter-rouge">drush cim -y</code></li>
<li><code class="language-plaintext highlighter-rouge">drush core:cron -y</code></li>
<li><code class="language-plaintext highlighter-rouge">drush cr -y</code></li>
<li><code class="language-plaintext highlighter-rouge">npm -C /app/web/themes/custom/$theme run build:dev</code> (This last command will vary depending on whether you compile your theme with something like Webpack or Gulp.)</li>
</ol>
</li>
</ol>
<p>You can see why we turned these into commands. I can’t tell you how many times I have seen one of these steps skipped and the resulting build just fails. Usually from a configuration conflict or an entity update that didn’t happen as it should.</p>
<h3 id="once-you-have-updated-your-composerjson">Once you have updated your composer.json</h3>
<ol>
<li><code class="language-plaintext highlighter-rouge">lando cupex</code> (This is kinda a goofy little shorthand I came up.)
<ol>
<li><code class="language-plaintext highlighter-rouge">composer update</code> (Yes, I just update it all and I do so pretty regularly.)</li>
<li><code class="language-plaintext highlighter-rouge">drush updb -y</code> (Runs all your update and post-update hooks which changes the database.)</li>
<li><code class="language-plaintext highlighter-rouge">drush cex -y</code> (This is the shorthand for config-export and it takes all those database changes and writes them to config.)</li>
</ol>
</li>
<li><code class="language-plaintext highlighter-rouge">gaa</code> (<code class="language-plaintext highlighter-rouge">git add -A</code> is my preferred way to deal with automation like this as it just grabs every change and I can see those changes in my editor.)</li>
<li><code class="language-plaintext highlighter-rouge">gcmsg "My commit message"</code> (This command is so much less verbose and easier to type than <code class="language-plaintext highlighter-rouge">git commit -m "My commit message"</code>. We include our Jira issue IDs in the message to tie it all back to our sprint board.)</li>
<li><code class="language-plaintext highlighter-rouge">git push -u origin $branch-id</code> (I think there is shorthand for this that I just haven’t taken the time to learn. Yeah… me too.)</li>
</ol>
<p>And here is where the magic happens and we let our GitHub and CircleCI integration take over and build our multidev environment for automated (Behat) and manual testing. QA still catches a lot, so we haven’t stopped doing it. Our CircleCI scripts are pretty much using the same set of commands to build our artifacts that get pushed to our servers on Pantheon.</p>
<h2 id="it-took-six-smaller-builds-to-finally-get-to-87">It took six smaller builds to finally get to 8.7</h2>
<p>You read that correct. I had to incrementally update a lot of “small” things and get all those patches to apply cleanly before the final build would give me a site that had no errors and passing tests. That means I had to repeat the “<em>latest</em> > <em>refresh</em> > <em>change some things</em> > <em>cupex</em> > <em>push to origin</em>” over and over. Developing in PHP is so much like developing in Javascript now! Write some recipes and compile! Sigh.</p>
<p>Funny story… after working on this upgrade off an on for a month, and having a build that worked in our dev site, our build process was halted by a system outage upstream. <code class="language-plaintext highlighter-rouge">¯\_(ツ)_/¯</code>.</p>
<h2 id="was-it-worth-it">Was it worth it</h2>
<p>In truth, yes. I learned a lot about Drupal 8 in this process. Over the last year of this project with the City of Portland, I have learned just how much of my D7 and D6 knowledge applies and just how much does not. I am much more intimately aware of some of the subsystems now than I was before—even if I still don’t consider myself much of a true PHP developer as much as a really advanced site builder with some frontend chops.</p>
<p>Drupal 8.7 has several cool new features that make it worth trying. In fact, on a clean install, I found a lot to love by grabbing Lightning 4—released on May 16—and adding a minimal amount of configuration.</p>
<p>Chasing core and staying up to date with your dependencies as feasible is important with Drupal. In truth, this is important with any software now as dependency management and compiling a lot of a code written by a lot of different people is a must to build the complex tools that we try to make look simple to the people that use our software.</p>
<p>Did I get any of the above completely wrong? Have a question I didn’t answer? Hit me up in the comments. I haven’t seen a comment in months.</p>
Questions and answers from Oregon Digital Government Summit2019-06-20T00:00:00+00:00http://joshuami.com/2019/06/20/oregon-digital-government-summit-citizen-centric<p>I was a part of a panel at the Oregon Digital Government Summit in Salem, Oregon. Here are a few of the questions I was asked and my responses during the panel “Becoming Citizen-Centric”.</p>
<h2 id="question-how-do-you-make-sure-your-sites-are-citizen-centric">Question: How do you make sure your sites are citizen centric?</h2>
<p>Answer: It starts with planning. Our vision is to create a site where services and information are easy to find, easy to access, and easy to understand. Properly typing your content and applications is key to helping people find what they need. You have to train your search engines. Accessibilty is not just an equite issue. Everyone is differently abled at one point in their life. It could be an arm injury or gradual loss of visual acuity with age. It could be cognitive issues caused by injury or even stress. Accessibiliy is also about device differences. Over 50% of our public traffic is mobile—that means every other person is getting their information on a small screen. We have to build for that from the start. Finally, we have to think about how easy our services are to understand. We aim for a reading level of 5th to 8th grade in our content and we are training editors how to check their work before they publish.</p>
<h2 id="question-what-tools-help-with-planning-a-citizen-centric-site-or-applicaiton">Question: What tools help with planning a citizen-centric site or applicaiton?</h2>
<p>Answer: We use a wide range of tools. I personally like to start with analytics and any data that describes our users in broad categories. Then, we break down broad metrics into smaller measurements of success that is compared to personaes that we create for types of community members.</p>
<p>To make sure we are not missing a point of view, we do pretty exhaustive usability testing of our tools after they have been built and respond to that feedback in our products.</p>
<h2 id="question-you-mention-findability-how-do-you-make-sure-things-are-easy-to-find-on-your-site">Question: You mention findability? How do you make sure things are easy to find on your site?</h2>
<p>Answer: People use two techniques to get to content; they browse and they search. Browsing is about a really solid information architecture that uses words our community understands. People don’t care about our org charts. They don’t care about the bureaucracy. They just want to finish a task as quickly as possible.</p>
<p>Search is a little trickier. Too many sites implement a search appliance or an application and do nothing to train that application to understand the underlying content. Here is an example. For the city of Portland, parks, park, parked, parking all have slightly different meanings in search. The first two should deliver information about our Parks and Recreation programs. The last two should deliver information about parking tickets and parking meters. Out of the box, a search engine will treat all of those versions of the word “park” as the same thing. We have to train our search engines and that is a lot of really detailed work that is more art than purely science.</p>
Portland.gov Project Overview2019-05-05T00:00:00+00:00http://joshuami.com/2019/05/05/portlandgov-project-overview<p>At DrupalCon Seattle, I presented <a href="https://events.drupal.org/seattle2019/sessions/content-structures-government-websites-build-plan-portlandgov">Architecting government websites: migrating Portland.gov to Drupal 8</a>. Unfortunately, the 90-minute sessions were not recorded this year. I have had a lot of requests for more information about the project with Portland and the <a href="https://events.drupal.org/sites/default/files/slides/Migrating%20Portlandgov%20DrupalCon%20Seattle2019%20compressed.pdf">slides from the presetation</a>.</p>
<p>We posted an overview of the project on our beta site. <a href="https://beta.portland.gov/powr/about">About the POWR project</a>—we do love our acronyms in government. This overview takes a high-level walk through the project and its objectives. A migration of this scale is <strong>not a small project</strong>, but the eGovernment team at the City of Portland have been amazing—and a ton of progress has been made. The hardest part of an effort like this is the wrangling of stakeholders across over 30 offices and hundreds of existing editors.</p>
<p>It still feels like we are on track for a 2019 switchover of <a href="https://www.portlandoregon.gov/">PortlandOregon.gov</a> over to the new <a href="https://portland.gov/">Portland.gov</a>. I cannot wait to see how the public responds to the new site when we start marketing the switch!</p>
Microsoft's anti-collaboration suite2018-09-23T00:00:00+00:00http://joshuami.com/2018/09/23/microsofts-anti-collaboration-suite<p>After over 7 years of primarily using Google’s suite of productivity tools, I recently found myself thrown back into a Microsoft-first world. It has not been a pleasant experience.</p>
<p>I was once a Microsoft power user. I could build Word and Excel documents with the best of them. And Outlook… I had all the keyboard commands memorized and was an early adopter of Inbox Zero. Arguably, the product has changed quite a bit since that time—circa Windows XP.</p>
<p>Summing up the biggest difference in collaboration between Microsoft Office365 suite compared to more polished offerings like G Suite, Microsoft assumes files will be passed back and forth between users while Google assumes that you will collaborate in real time. This is a huge difference.</p>
<p>Microsoft has not updated the basic premise of Office, and related tools such as Sharepoint, in over 15 years. They may feel they don’t need to change due to marketshare or that Office365 is an adequate response to online-only suites like Google’s—but neither would be the case.</p>
<p>Microsoft’s marketshare is erroding quickly in small to medium sized businesses that are looking for a simpler maintenance and licensing model without expensive IT support. Even slower to change organizations at the enterprise level are responding to their power users that insist on better collaboration tools. These organizations are looking at alternatives to the Microsoft suite for file sharing and collaboration.</p>
<p>I don’t want a single company to own a monopoly over every aspect of our online life. I am supportive of Microsoft being able to compete in this space, but they are going to have to work for it.</p>
<p>What would make Office365 more competitive tool for collaboration?</p>
<p>Here are my suggestions…</p>
<h2 id="simplify-your-tools">Simplify your tools</h2>
<p>I love Google’s suite of productivity tools. They are simpler when they need to be—there are no maddeningly complex “ribbons” to contend with—and they are much more stable when collaborating. I remember Office before it adopted the ribbon. It was easier to use. I’m not sure what user research drove Microsoft to the ribbon, but I think the testing had to have been flawed—not just because G Suite is easier for me to use. Microsoft has lost marketshare to simpler productivity tools over recent years.</p>
<p>As a comparrison, look at the G Suite Docs experience versus Word experience in Office365.</p>
<p><img src="/public/images/microsoft-word-complexity.png" alt="Microsoft Word has added so many features that it has become virtually unusable." /></p>
<p><img src="/public/images/google-docs-simplicity.png" alt="G Suite's Docs application is limited to the features you need to start and progressively exposes advanced features to power users." /></p>
<h2 id="settle-on-a-model-that-is-saas-based-and-api-first">Settle on a model that is SaaS-based and API-first</h2>
<p>Heck, even <a href="https://en.wikipedia.org/wiki/Satya_Nadella">Satya Nadella</a> wants to see a seamless experience between all devices. And yes, OneNote is pretty cool and effective for a single person to sync their notes between all their devices, but SaaS-based collaboration would be even better.</p>
<p>Microsoft office has a dizzying array of product offerings. Are you getting your product for “home” or “office” or “business” or your “team”? Why is there different pricing for “firstline workers”? What even is a “firstline worker” in the context of productivity tools? There is a different price point for each that contains a different set of software. (Some of the software listed is even free if downloaded outside of Office.)</p>
<p>Whether trying to deciper their menu…
<img src="/public/images/microsoft-office-products-sept-2018.png" alt="Microsoft's menu for Office products as of September 2018." /></p>
<p>… or the home options…
<img src="/public/images/explore-all-office-products-sept-2018.png" alt="Explore all Office products for home use as of September 2018." /></p>
<p>… or the business options…
<img src="/public/images/office365-for-business-pricing-sept-2018.png" alt="Busines products available through Office365." /></p>
<p>… or the enterprise options…
<img src="/public/images/office365-enterprise-pricing-sept-2018.png" alt="Enterprise products avaiable through Office365." /></p>
<p>… Microsoft really wants you to pick a package that is tailored to their business model more than the solution is tailored to your needs as a business or consumer.</p>
<p>Stop the sales insanity! It doesn’t make the product better or more focused.</p>
<p>Microsoft should simplify their product line around pricing that is suite-based and includes all the tools that you can reasonably offer with an online interface. If you want to include a free tier, feel free. Getting users to buy into using Microsoft Office at home makes a lot of sense for getting them to buy into the product in their business.</p>
<p>A truly SaaS-first product should focus on its API to increase interoperability. You should be able to access every major feature in online software even if there is a desktop version. If this is not possible, cut back on features.</p>
<p>Office is far too bloated at this point. I see far more examples of poorly-crafted and inaccessible documents made with Word and Powerpoint and Publisher than cleverly-used advanced features.</p>
<h2 id="improve-the-online-editing-experience">Improve the online editing experience</h2>
<p>Would you like to download the software or use it online—well, Microsoft would really like you to use the installed softare even if you are using it with their online account offering.</p>
<p><img src="/public/images/edit-in-word-versus-edit-in-browser.png" alt="Their online interface shows edit in Word as your first option with edit in 'browser' as the second option." /></p>
<p>Why do we even need options for editing in the online experience? Just let me edit and get out of my way?</p>
<h2 id="simplify-your-product-line">Simplify your product line</h2>
<p>Why is the experience of OneDrive, Sharepoint, and Teams different? Depending on which of these tools are used, your experience of “sharing” a document for collaboration is completely different. Make a decision on the direction Microsoft wants to take and stick with improving that experience for your software as a service users.</p>
<p>What is the advantage of splitting development between Skype for Business (a poorly executed rebranding of Lync)?</p>
<p>Microsoft should pick one name for file synchronization and stick with it. They should make it clear how much each account has used and make it particularly easy to share within and outside an organization.</p>
<h2 id="optimize-revisioning-for-collaboration">Optimize revisioning for collaboration</h2>
<p>Whether Microsoft intends it or not, their platform encourages users to create revision after revisions in shares and directories resulting in files like \some-share\some-crazy-directory-structure\some-document-name_rev22_final_bob.docx. My favorite is when you have revisions from three different collaborators that are all labeled “final”.</p>
<p>Semantic versioning could be a solution, but that still means duplicating everything in the document and tracking which version of the document is currently the latest.</p>
<p>Here’s the thing, you don’t need to change the label of your file if you software exposes the revisioning layer in a simple and understandable way.</p>
<p>Track changes was revolutionary for providing this experience—thank you, Microsoft!</p>
<p>Microsoft made track changes the default go to for redlining documents in the business world. Now, they should optimize for track changes to feel more like the Google equivalent with suggestions that can have comments and resolution. Google truly improved this foundational productivity feature.</p>
<p><img src="/public/images/google-docs-killer-collaboration.png" alt="The G Suite Docs suggesting mode is track changes, but with inline comments and notifications." /></p>
<p>Google’s approach is currently collaboration at it’s best.</p>
<p>Microsoft should feel free to steal back Google’s innovation—assuming patent law allows. It is kinda how this whole software as a service thing works. Am I right?</p>
<h2 id="storage-should-be-integrated-with-the-cloud-and-seamless">Storage should be integrated with the cloud and seamless</h2>
<p>Storage is not adequately integrated in the Microsoft world. OneDrive is buggy and confusing compared to Dropbox, Box, or even Google Drive. Sharepoint seems to be closed to only your organization—unless you expose it as FTP?!—which I suppose is good if you are trying to protect documents, but it is all or nothing for users. There are no solid ways to share with contractors and partner organizations outside of your organization’s domain.</p>
<p>Office365 was supposed to compete with G Suite, but it really doesn’t allow for any of the ease of search or web-only storage that makes G Suite so easy to roll out and support.</p>
<p>Dropbox may have the best example of seemless sharing of all the storage options. Sync is about the same across each of these solution, but Dropbox allows the publishing of a space where others can download the documents you’ve shared publicly. It’s a nice touch that this can be used to help extend access to vendors and collaborators outside of your organization that do not need editing rights.</p>
<h2 id="offer-a-free-tier-to-build-a-base">Offer a free tier to build a base</h2>
<p>Perhaps the most compelling feature of Google’s suite of productivity tools is that you can have them for free—with a pretty decent amount of storage included.</p>
<p>Office365 needs a tier of users that get all the benefits of Word and Excel without having to plunk down a chunk of change. $99 a year is a lot to get a ton of features that you’ll never need with a poorer email experience and sad collaboration tools. To compete, Microsoft is going to need to try and level the playing field to gain new users.</p>
<h2 id="not-about-good-or-bad">Not about good or bad</h2>
<p>Microsoft is not “bad” or “good”, but they need to improve and keep up with the current demands of consumers and businesses.</p>
<p>I am not a Microsoft hater. While it is true that I prefer MacOS or Linix, I’m actually a huge fan of a couple of Microsoft’s products. Visual Studio Code is one of the most impressive IDEs that I’ve ever used. (And yes, I have used a lot over the years.) OneNote is an amazing note taking tool on both desktops and tablets. I was once a power user in every sense on a Windows PC. I cut my teeth developing in a Microsoft-centric world.</p>
<blockquote>
<p>Microsoft is not “bad” or “good”, but they need to improve and keep up with the current demands of consumers and businesses</p>
</blockquote>
<p>Microsoft has the resources to be better.</p>
<h2 id="but-i-worry-about-products-microsoft-has-ruined">But… I worry about products Microsoft has ruined</h2>
<p>Skype is nowhere near the product it was before Microsoft took over. Yammer was also better before Microsoft took the wheel.</p>
<p>Side note: please, please for the love of all that is sacred and open source, don’t ruin GitHub!</p>
<h2 id="my-hope">My hope</h2>
<p>Microsoft—and I speak to you as if you are anything close to a single entity—please get this right. Give us options and continue to create a competitive market with Google. Frankly, if you don’t, other companies are unlikely to do so. Worse, you will continue to see your marketshare diminish and consumers will be left with only a monopoly. Sound familiar? It has happened before.</p>
React's Switch to the MIT License is Important2017-09-27T00:00:00+00:00http://joshuami.com/2017/09/27/why-react-mit-license-is-important<p>Kudos to Facebook for listening to their community!</p>
<p>Just this past week, Facebook decided to <a href="https://code.facebook.com/posts/300798627056246/relicensing-react-jest-flow-and-immutable-js/">relicense React, Jest, and Immutable.js</a> to use the MIT license instead of the BSD derivative with troubling patent clauses that was their licensing choice only a few weeks prior.</p>
<p>There were likely a couple of different actions in the open source world that triggered this decision.</p>
<p>First, in August, the Apache Software Foundation categorized the Facebook BSD+Patents license as a Category-X license. Essentially, they were declaring it incompatible with Apache licensing because of the patent clauses.</p>
<p>Second, and probably much more impactful, was the <a href="https://ma.tt/2017/09/on-react-and-wordpress/">decision by the Wordpress project to move away from React due to its licensing</a>. I’m curious to see if that leads to the Wordpress community sticking with React in the Calypso and Guttenberg projects. It would certainly save a lot of rework, but in watching the <a href="https://www.drupal.org/node/2645250">Drupal community struggle with the decision to include a javascript framework</a>, I can imagine that there will still be a strong push for projects like Vue, Preact, or Polymer to get included.</p>
<p>A while back, I published a series of posts on open source licensing. In <a href="https://www.phase2technology.com/blog/open-source-licensing-part-3-which-license-should-i-choose/">part 3 of that series</a>, I tried to direct people to possible licenses for their project. I am particularly keen on MIT.</p>
<blockquote>
<p>I think MIT is the simplest and easiest license in terms of readability. MIT is a copyright statement and 3 paragraphs of simple legal text to allow people to use your code as they see fit so long as they provide some attribute back and do not hold you liable. It is the most used license on Github and is preferred by most Node projects. Technically, “MIT license” is a little ambiguous. For clarity, I’m referring to what some call the “Expat License” because it doesn’t include a trademark statement making it slightly more permissive and allowing it to be as compatible with other licenses as possible.</p>
</blockquote>
<p>I still think that the MIT expat license is the most human readable—as opposed to lawyer readable—of the open source licenses. It pretty much just states that people are free to use your code without a lot of fuss about what you plan to do with it or how you distribute it afterwards.</p>
<p>The MIT license is simple enough that it does not scare away businesses looking to keep a handle on their open source software portfolio. It lacks the aggressiveness of protective copyleft licenses like the GPLv3 that many businesses seek to avoid. Whether a business needs to avoid the GPLv3 for fear of losing rights on proprietary software they have created is not an issue one wants to confront when trying to build a great solution for your client or organization.</p>
<p>Facebook made a good choice for the communities of developers that use their code—even if that decision may have left their lawyers with concerns about petty litigation. It is heartening to see open source market forces work out for the greater good.</p>
A Layperson's Guide to Open Source Licensing2017-04-06T00:00:00+00:00http://joshuami.com/2017/04/06/guide-to-open-source-licensing<p>This post was republished from a series originally contributed to the <a href="https://www.phase2technology.com/blog-list/">Phase 2 blog</a>:</p>
<ul>
<li><a href="https://www.phase2technology.com/blog/open-source-licensing-part-1-an-introduction/">Open Source Licensing Part 1: An Introduction</a></li>
<li><a href="https://www.phase2technology.com/blog/open-source-licensing-part-2-software-licensing-is-a-continuum/">Open Source Licensing Part 2: Software Licensing is a Continuum</a></li>
<li><a href="https://www.phase2technology.com/blog/open-source-licensing-part-3-which-license-should-i-choose/">Open Source Licensing Part 3: Which License Should I Choose?</a></li>
</ul>
<p>I was recently asked to provide some feedback on a couple of projects that Phase2 is planning to release under an open source license. As a company, we definitely have a bias towards using open systems because they give our clients flexibility, cost savings, and transparency. Picking an open source license is complex, but there are reasons for choosing one license over another.</p>
<h2 id="disclaimer">Disclaimer</h2>
<p>The following topic is complicated. I’m offering up my research, but also need to offer up the following disclaimer. I am not a lawyer. I am a technologist. While I have a pretty extensive history in using and participating in open source projects, this work should not be taken as legal advice. That said, I do not think just any lawyer will do to help you decipher your open source licensing needs.</p>
<p>If you are truly interested in the legal ramifications of a software licensing decision you need to make, consult with a lawyer that specializes in copyright, intellectual property, and software law. Further, software licensing and the related laws differ from country to country based on the legal systems present. Make sure your legal advice comes from someone that knows law that covers your type of software. Software licensing for devices, particularly networked devices, differs significantly from web and service software, which will be the focus of this post.</p>
<h2 id="what-is-open-source">What is open source?</h2>
<p>Open source is a huge concept. At its simplest level, open sourced software has not been compiled. A person can analyze the source code that allows the application to run. Open source does not mean free, though many open source projects are also FOSS (free open source software). It is worth noting that one of the founders of the free software movement, Richard Stallman, would argue that open source “<a href="https://www.gnu.org/philosophy/open-source-misses-the-point.html">misses the point</a>” of software truly being “free” to use.</p>
<p>The original intent of much of the FOSS movement was to make software free for use and for developers to extend the uses of software. As a movement, it has morphed into legal protections to allow either the developer or the owner of the intellectual property created or the user of the finished software to maintain or release rights to that work. The tool used to accomplish this declaration of rights is a license.</p>
<h2 id="what-is-a-license-and-why-does-it-get-applied-to-software">What is a license and why does it get applied to software?</h2>
<p>A software license is an intersection of copyright, patent law, and legal precedents. Since software is essentially written word, in most countries, it falls under copyright law. Most modern copyright law in the United States was established in the Copyright Act of 1976. The STELA Reauthorization Act of 2014 extended the bulk of this law. The recognized power to establish copyright law—and patent law—stems from the constitution itself. (Article 1, section 8)</p>
<p>Perhaps it is because people in the United States are litigious, but the US is where most open source licensing law has been established and tested—albeit tested in limited fashion, which I will get into a little later.</p>
<p>The <a href="https://www.copyright.gov/title17/92chap1.html">scope of copyright</a> is pretty well established.</p>
<blockquote>
<p>“Copyright protection subsists, in accordance with this title, in original works of authorship fixed in any tangible medium of expression, now known or later developed, from which they can be perceived, reproduced, or otherwise communicated, either directly or with the aid of a machine or device.”</p>
</blockquote>
<p>Technically, anything original that is written gets the protection of copyright given to the author or authors of that work. This protection extends for the life of the author or authors, plus a default of 70 years. Yes, in a work by more than one author, copyright extends to all the participants equally. This joint “ownership” becomes extremely important in open source licensing.</p>
<p>The license itself is a legal statement by the owner of the copyright to share their intention for that work.</p>
<p>Lest you have any doubt about it, unlicensed software with no mention of copyright leaves that software in the public domain. Everyone can use it and distribute it as they wish—which sounds great—except that a license could still technically be applied to this work at a later time by the author or authors of the work.</p>
<p>Lack of a license leads to a lot of confusion as to what the intent was for a large body of work in distributed code hosting services such as Github. That confusion of intent was an impetus for starting choosealicense.com—a Github project that helps new project creators choose a license and apply it to their project.</p>
<p>If you are releasing software for others to use, please add a license—especially if you intend for others to share in the creation of that software on Github or Gitlab or even a self-hosted versioning repository that others can access.</p>
<h2 id="open-source-licensing-is-a-continuum">Open source licensing is a continuum</h2>
<p><img src="public/images/open-source-licensing.png" alt="Representation of the open source licensing continuum" /></p>
<p>On one end of the continuum is public domain. Just to the right of the public domain are permissive licenses like MIT and BSD. The solid middleground of software licensing is protective, the most common of which are the GPL and Apache licenses. Even within the released versions of those licenses, there are important distinctions that put them on a continuum with each other. The GPL is decidedly Copyleft—meaning it has provisions to ensure that people contributing to it license the derivative work as GPL as well.</p>
<p>Just to the right of protective FOSS licenses are freeware or shareware licenses that were wildly popular ways to pass on desktop applications for a time but have faded as other licenses become more common. Proprietary licenses fall solidly on the most conservative edge of the license spectrum. There is no further to go after proprietary than keeping the software a trade secret.</p>
<h2 id="where-do-patents-come-in">Where do patents come in?</h2>
<p>It gets a little more interesting when you consider what copyright does not protect.</p>
<p>In no case does copyright protection for an original work of authorship extend to any idea, procedure, process, system, method of operation, concept, principle, or discovery, regardless of the form in which it is described, explained, illustrated, or embodied in such work.</p>
<p>That provision is where we get into patent law and the protection of intellectual property. Patents can’t really be applied to just an idea. Patents can only be applied to the description of a tangible object in order to protect that object from creation or distribution by others.
Patents come into play with software when the software is integrated into the object/device/machine being patented. Some licenses have patent clauses that explain contributors rights to use the patent for the purpose of development without giving up the right to protect the final product.</p>
<h2 id="license-compatibility">License compatibility</h2>
<p>While challenges to open source license precedent are rare, one area where there has been a ton of review and interpretation is compatibility. The language of a license can naturally lead to a conflict with another license if works are combined into a derivative. Nearly all compatibility concerns have to do with the distribution of the final software derivative.</p>
<p>In general, permissive licenses (MIT, BSD) can be combined into more protective licenses (LGPL, GPL) with the derivative getting licensed as a whole under the more protective free open source software license. There are even licenses especially designed to allow for a derivative that is a combination of proprietary and protective open licensing. The LGPL is a good example of this compromise to help protect business interests while still open sourcing significant libraries used within a larger framework.</p>
<p>A couple of important exceptions to compatibility pattern can be seen in combinations such as Apache2 (a permissive license with wording to protect patent use) and the GPL version 2 (a strongly protective license). These two licenses have conflicting clauses that make them incompatible with each other.</p>
<p>Some of the biggest open source projects (Drupal, Wordpress, Linux) have gotten around this incompatibility by using the GPLv2, but expressly stating that it is okay for the project to be combined with any later version of the GPL for distributions. This is sometimes referred to as GPLv2+ or GPLv2 and later. Do not let the common names for this confuse, those projects are released as GPLv2.</p>
<h2 id="why-choose-a-license-from-1991">Why choose a license from 1991?</h2>
<p>The second version of the GPL was released in 1991. That is forever ago in technology years. It was one of the first and most important “copyleft” licenses. Copyleft is the protective part of the GPL. It has intentional provisions to force the resulting distribution to be free and open source. That makes it very opinionated.</p>
<p>The GPLv3 was released in 2007—still an eternity—but it was intentionally written to be compatible with software that is GPLv2 or Apache2 that is being combined into a GPLv3 licensed derivative. This sounds great, but as a result of the GPLv3 being fiercely protective and Copyleft, many lawyers and companies are hesitant to introduce this license into derivative works in their software portfolio.</p>
<p>Wordpress and Drupal both predated the Apache2 license (released in 2004), and chose to keep the GPLv2—which was chosen based on the Linux kernel’s license of choice. The result is that if you want to host a Drupal module on Drupal.org or a Wordpress plugin on Wordpress.org, you are releasing that module under GPLv2. That means the combined code for the module could include libraries that were licensed under more permissive licenses, such as MIT, but those libraries cannot be either Apache2 or GPLv3 as provisions in those licenses are incompatible. Technically, you could create a derivative of Drupal or Wordpress and some related modules or plugins that were GPLv3, but you would not be able to host them in the repositories those projects host. Complicated, eh?</p>
<h2 id="what-have-composer-and-npm-done-for-compatibility">What have composer and NPM done for compatibility?</h2>
<p>Both NPM (node.js) and Composer (PHP) provide some compatibility checking by allowing projects to clearly identify their license.
Further I would argue, and here is where you should stop trusting me and talk to a lawyer, that a composer.json or package.json requirement does not represent distribution since the code is not packaged into a combined tar or zip. Rather, those files are combined on the server via scripts that request the packages separately assembling the final package. Therefore the final product running on the server would not constitute a derivative since it will not be distributed.</p>
<p>I could be wrong in my interpretation, and I encourage legal feedback, but if project requirements represent “a derivative” that can be “distributed” then much of the open source world is out of compliance with a great many implementations of open source that are defaulting to the most protective license—likely GPLv3.</p>
<p>I work with a lot of free open source software; the value does not come from the code but from the services provided on top of that code. However, incompatible combinations do leave corporations exposed to possible lawsuits. Consult a lawyer to review your licensing and options if you are concerned.</p>
<h2 id="which-license-should-i-choose">Which license should I choose?</h2>
<p>There is no easy answer to that question, but I can give some general guidelines that are a little more opinionated that what Github provides with choosealicense.com.</p>
<h3 id="want-people-to-use-your-code-choose-mit-first">Want people to use your code? Choose MIT first.</h3>
<p>I think MIT is the simplest and easiest license in terms of readability. MIT is a copyright statement and 3 paragraphs of simple legal text to allow people to use your code as they see fit so long as they provide some attribute back and do not hold you liable. It is the most used license on Github and is preferred by most Node projects. Technically, “MIT license” is a little ambiguous. For clarity, I’m referring to what some call the “Expat License” because it doesn’t include a trademark statement making it slightly more permissive and allowing it to be as compatible with other licenses as possible.</p>
<h3 id="are-you-concerned-about-patents-consider-apache2">Are you concerned about patents? Consider Apache2.</h3>
<p>Apache2 is big in the mobile world because of its patent provisions. If your code is going to run on a device where you think you might need to be concerned about patent infringement or misuse by your developers, then Apache2 is a good solution. Android uses it. Swift uses it. Google and Apple probably knew what they were doing to protect their hardware patents while still encouraging a diverse and active developer ecosystem. The only major catch with the Apache2 license is that the resulting code will not be compatible with GPLv2 code from a derivative distribution standpoint.</p>
<h3 id="are-you-developing-a-module-for-drupal-or-a-plugin-for-wordpress-use-gplv2">Are you developing a module for Drupal or a plugin for Wordpress? Use GPLv2.</h3>
<p>I know what you are thinking. Why would I use the license from 1991 for my shiney new project? The short answer is you have to if you want people to find your code and use it. You could host your project on Github, Gitlab, Bitbucket, etc and choose something different, but your code will be more likely to be used if it is in the project repositories of the parent projects.</p>
<p>You can include libraries in your module or plugin that contain compatible permissive licenses (MIT, BSD), but you cannot include incompatible libraries (Apache2, GPLv3, etc.). If you do, those modules will be removed as not compliant with the terms of service. (As soon as someone realizes it is there… that is a whole other story.)</p>
<h3 id="what-about-gplv3-or-lgplv3-or-insert-license-name-here-my-project-wants-freedom">What about GPLv3 or LGPLv3 or [insert license name here]? My project wants freedom!</h3>
<p>I get it. There are reasons, particularly for using LGPL to allow for compatible licensing with proprietary components, to use these licenses. The glibc (de facto C library for Linux distributions) used to LGPL in order to ensure adoption. The AGPL has some provisions that make it better for networked solutions if you want to ensure that modified versions of your code are shared under the same license. MongoDB for instance is licensed under a combination of the AGPLv3 its server and tools and Apache2 for its drivers—and just to make it even more interesting there is a commercial license option as well for companies that feel the AGPL is too protective for their lawyer’s comfort.</p>
<p>Gnu.org has one of the most <a href="https://www.gnu.org/licenses/license-list.html">comprehensive lists of licenses and their compatibility with the GPL</a> that I’ve found . It’s a great starting point for research if you are picking a license outside of the options covered in this document.</p>
<p>I am not that opinionated about license options. I prefer to focus on developing the solution and then making sure I can explain the implications in distributing what was created.</p>
<p>Because of its Copyleft provisions, many feel that eventually a lot of software is going to have to be released as GPLv3 to remain compatible. That doesn’t mean it has to start that way if it is not the best choice for your business.</p>
<h2 id="software-licensing-is-not-content-licensing">Software licensing is not content licensing</h2>
<p>Guess what! Your content is not covered by the licenses I have covered here. I mean, you could argue that documentation within the code is covered by the software license, but if you built a content management system (CMS… you know Wordpress, Drupal, etc) that content is not covered by the software license. The copyright for that content belongs to the owner. The owner is often defined by the terms of service or end user license agreement for the site or service.</p>
<p>If you want to make your content open, there are great tools. A good starting point is <a href="https://creativecommons.org/">Creative Commons</a>. Share your work… define how others can use it.</p>
<h2 id="make-your-own-license-please-dont">Make your own license! (Please don’t)</h2>
<p>You are probably thinking you could write an even better license than all those lawyers that created the compatibility morass that is stopping your work from being free!</p>
<p>You probably cannot. I am pretty sure you should not try, but hey, I cannot stop you.</p>
<p>Happy sharing.</p>
<h2 id="addendum-resources">Addendum: Resources</h2>
<ul>
<li>https://www.smashingmagazine.com/2010/03/a-short-guide-to-open-source-and-similar-licenses/</li>
<li>https://www.copyright.gov/title17/92chap1.html</li>
<li>https://opensource.org/licenses/MIT</li>
<li>https://ttmm.io/tech/why-the-mit-license/</li>
<li>https://www.gnu.org/philosophy/open-source-misses-the-point.html</li>
<li>http://www.fsf.org/</li>
<li>https://www.gnu.org/licenses/license-list.html</li>
<li>https://www.gnu.org/licenses/old-licenses/gpl-2.0.html</li>
<li>https://www.apache.org/licenses/LICENSE-2.0</li>
<li>https://www.gnu.org/licenses/licenses.html</li>
<li>https://en.wikipedia.org/wiki/Software_license</li>
<li>https://opensource.org/licenses</li>
<li>https://www.uspto.gov/</li>
<li>https://www.copyright.gov/</li>
<li>https://choosealicense.com/</li>
<li>https://www.gnu.org/licenses/rms-why-gplv3.en.html</li>
<li>https://www.kernel.org/doc/pending/gplv2-howto.html</li>
<li>https://www.linux.com/news/gpl-requirement-could-have-chilling-effect-derivative-distros</li>
<li>https://creativecommons.org/</li>
<li>http://ben.balter.com/2015/06/03/copyright-notices-for-websites-and-open-source-projects/</li>
</ul>
Water2017-01-10T00:00:00+00:00http://joshuami.com/2017/01/10/water<p>You truly do not know how precious water is until it is scarce.</p>
<p>Normally, I only feel that bit of water-related dred on camping trips when my family and I find ourselves having to meter out water in order to avoid an extra trip to the well. I have yet to truly rough it along a river and use purification tablets or a filter to ensure the water was potable. As a long time resident of the rainy part of the Northwest, I seldom go long without water surrounding me in some fashion. My family and I are fortunate. We do not want for water.</p>
<h2 id="practice-emergency">Practice emergency</h2>
<p>Starting yesterday and lasting into the morning hours today, my family got to experience what it would be like to live on emergency water. It was a bit of a practice should we ever have a major disaster—natural or manmade—befall us.</p>
<p>Having your sprinkler system—see water everywhere and we still have irrigation systems for our moderate summers—freeze during an unusually cold winter can lead to sudden inability to control that system when it thaws again. Having not had this experience before, I didn’t know where my shut off valve was for the system, so I shut the water off at the meter cutting water to the entire house.</p>
<p>While it was warm enough to begin to thaw, the streets were still icy enough that landscape crews were not available to come out and help us find the irrigation valve, so we kept the water shut off and began our “practice emergency” with about 6 gallons of potable water.</p>
<h2 id="your-children-are-not-prepared-mine-might-be">Your children are not prepared… mine <em>might</em> be</h2>
<p>Six gallons should be enough to get through a day for four people, a dog, a cat, and two chickens. That said, we were starting off our practice emergency from a deficit. According to <a href="https://www.ready.gov/water">Ready.gov</a>, you should have one gallon of water per person per day, and at least a three day supply, to be ready for a emergency.</p>
<p>That would mean we should have started our practice emergency with at least 12 gallons of water for the people in our home. I need to look into how much the animals would need. Fortunately, they are not quite as picky as us humans. I’ve seen what my dog will eat and drink—it’s not pretty.</p>
<p>That said, we might have been closer to the prescribed amount than we thought if we were to get creative. Points to my teenage son for coming up with a couple of the possible sources at our disposal:</p>
<ul>
<li>3 gallon jug we keep in the garage to be prepared and to use for camping</li>
<li>3 gallons of water bottles we keep in the basement for emergencies</li>
<li>20 gallon water heater, which could be drained in an emergency situation</li>
<li>6 gallons of water in our toilet reservoirs (Though I pointed out we would want to boil that no matter how clean that part of the toilet is supposed to be.)</li>
<li>1.5 gallons of ice in the freezer</li>
</ul>
<p>We also had some cans of La Croix (yes, I’m slightly addicted to the bubbly stuff), 2 or 3 bottles of juice, a half gallon of milk, maybe a quart of half in half as additional liquids on hand.</p>
<p>All told, that’s about 34 gallons (128 liters) of liquid to sustain us in an emergency.</p>
<h2 id="but-how-quickly-it-would-disappear">But how quickly it would disappear</h2>
<p>We knew we were likely getting water service back online in less than 24 hours. So we were frivolous. We used most of our large jugs of water for flushing toilets, washing up, brushing our teeth. Just knowing water was scarce made us a bit more desirous of it. All of us felt noticeably more thirsty.</p>
<p>Water was used quite quickly. By morning, we had polished off about 5.5 gallons of our jugs of water. 5.5 gallons gone in about 18 hours including 8 hours of sleep.</p>
<h2 id="emergency-preparedness-is-a-luxury">Emergency preparedness is a luxury</h2>
<p>The fact we had water on hand that could get us through a short emergency is a dream for many. Whether it be wars or drought, water is a scarce necessity to many. <a href="https://www.unicef.org/media/media_81329.html">Unicef recently reported that nearly 750 million people will without adequate drinking water.</a> That’s about 1 in 10 people in the world. <a href="http://water.org/water-crisis/water-sanitation-facts/">Water.org reports that 1 in 3 people worldwide lack access to a toilet.</a></p>
<h2 id="the-us-is-not-immune">The U.S. is not immune</h2>
<p>From Flint, Michigan to Portland Public School in Oregon to yearly droughts in California, the United States is not without its own water issues. Just last week, my kid’s winter concert and arts event had a stunning music and dance performance that focused on the issue of water and the experiences of “the water bottle generation”. Grant High School and many other schools in the area have been on bottled water due to unsafe lead levels found in the water system last year. The school district is still figuring out how to retrofit all these schools without a true budget option to do everything they should; for now, they will settle for doing the minimum they must.</p>
<h2 id="water-is-a-basic-human-right">Water is a basic human right</h2>
<blockquote>
<p>We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.</p>
</blockquote>
<blockquote>
<p>— Declaration of Independence</p>
</blockquote>
<p>While the U.S. constitution does not guarantee everyone access to clean water, it does suggest that “Life” is a basic right. I feel quite comfortable making the logical connection that water is life. Every human body is made up of water more so than any other compound.</p>
<p>In 2010, the <a href="http://www.un.org/es/comun/docs/?symbol=A/RES/64/292&lang=E">United Nations even adopted a resolution</a> stating that water is a basic “human right”. While that resolution has so many caveats as to be practically unenforceable, it at least stands as an attempt to recognize that humans must have water to survive.</p>
<p>Perhaps it is more than just a right. If we do not address this need, we risk violence and collapse of destabilized governments. The world is less safe without addressing basic human needs like water, food, shelter. If we tell people they can only have these basic needs met if they fight for them… then eventually they will fight.</p>
<h2 id="what-can-we-do">What can we do</h2>
<p>Support nonprofits that recognize the importance of the environment and water’s role in human rights. Give. Volunteer.</p>
<p>Conserve water. It is more than just shorter showers. Make sure to spend time every year with limited water usage in mind. It’s eye opening to camp or practice for an emergency.</p>
<p>Vote. Government is not the enemy of freedom. Push for better programs that do what they should do. You are lying to yourself if you think government programs make you less free. Poorly run programs are just that—poorly run—but there are fewer of those than some would assume. Most government programs actually give you important benefits that you would not have if you did not live with other people in a shared experience of location and belief systems. Laws are passed to protect that shared experience. That’s a good thing. That’s what nations should be.</p>
<p>Freedom is created by people living in a state of shared benefit. Access to clean water is a pretty powerful indicator of freedom.</p>
<p>You can’t have freedom by yourself. It must be shared.</p>
Starting a new phase at Phase22016-07-15T00:00:00+00:00http://joshuami.com/2016/07/15/starting-a-new-phase-at-phase2<p>In May, <a href="http://joshuami.com/2016/05/31/saying-goodbye/">I announced my departure from the Drupal Association</a>. Since my last day on June 10th, it has been a whirlwind of family events punctuated by conversations with organizations big and small about opportunities. There were so many cool places that I’ve talked to about what makes their work special. It was not an easy decision to pick a spot to land—it was even harder to end the month of extended family vacation. So much hiking and running and driving!</p>
<p>I’m excited to start at Phase2 as Director of Engineering on Monday, July 18th. I’ve long been a fan of Jeff and Frank and the incredible business and team culture they’ve built.</p>
<p>It’s always fun to start something new and get to know new coworkers. I can’t wait to meet my new clients and start getting to work adding value to their technology projects.</p>
<p>And now for a last bit of funemployment with family…</p>
DrupalCI - Continuous Integration Testing for Drupal.org2016-06-08T00:00:00+00:00http://joshuami.com/2016/06/08/drupalci-continuous-integration-testing-for-drupalorg<p><a href="https://www.drupal.org/drupalorg/blog/drupalci-continuous-integration-testing-for-drupalorg">Republished from Druapl.org</a></p>
<h2 id="why-test">Why test?</h2>
<p>The goal of automated testing is confidence: confidence in application stability, and confidence that new features work as intended. Continuous integration as a philosophy is about speeding the rate of change while keeping stability. As the number of contributing programmers increase, the need to have automated testing as a means to prove stability increases.</p>
<p>This post is focused on how the automated testing infrastructure on Drupal.org works, not actually writing tests. Much more detail about how to write tests during Drupal development can be found in community documentation:</p>
<ul>
<li><a href="https://www.drupal.org/simpletest">Testing (D7 and D8) / SimpleTest (D6)</a></li>
<li><a href="https://www.drupal.org/phpunit">Drupal’s implementation of PHPUnit</a></li>
</ul>
<h2 id="categories-of-testing">Categories of testing</h2>
<p>DrupalCI essentially runs two categories of tests:</p>
<p><strong>Functional tests</strong> (also called blackbox testing) are the most common type of test run on DrupalCI hardware. These tests run assertions that test functionality by installing Drupal with a fresh database and then exercising that installation by inserting data and confirming the assertions complete. Front-end tests and behavior driven tests (BDD) tend to be functional. <em>Upgrade tests</em> are a type of functional tests that run a full installation of Drupal, then run upgrade commands.</p>
<p><strong>Unit tests</strong> run assertions that test a unit of code and do not require a database installation. This means they execute very quickly. Because of its architecture, Drupal 8 has much more unit test coverage than Drupal 7.</p>
<p>These test categories can be broken down further into more specific test types.</p>
<h2 id="what-testing-means-at-the-scale-of-drupal">What testing means at the scale of Drupal</h2>
<p>Drupal 8, with its 3,000+ core contributors and 7,288 contrib developers (so far), needs testing as a means to comfortably move forward code that everyone can trust to be stable.</p>
<p>Between January and May 2016, 90,364 test runs were triggered in DrupalCI. That is about 18,000 test runs requested per month. Maintainers set whether they want tests to run on demand, with every patch submitted, or nightly. They also determine what environments those tests will run on; there are 6 combinations of PHP and database engines available for maintainers to choose from.</p>
<p>The majority of these test runs are Drupal 8 tests at this point. (19,599 core tests and 47,713 contrib project tests were run during those 5 months.) Each test costs about 12 cents to run on Amazon Web Services. At the time of writing this post, we averaged around $2,000 per month in testing costs for our community. (Thank you <a href="https://www.drupal.org/supporters">supporters</a>!)</p>
<h2 id="an-overly-simple-history-of-automated-testing-for-drupal">An overly simple history of automated testing for Drupal</h2>
<p>Automated testing first became a thing for Drupal contributed projects during Drupal version 4.5 with the introduction of the <a href="https://www.drupal.org/project/simpletest">SimpleTest module</a>. It was not until Drupal 6 that we started manually building out testbots and running these tests on Drupal.org hardware.</p>
<p>In Drupal 7, SimpleTest was brought into Drupal Core. (More information about what that took can be reviewed in the <a href="https://groups.drupal.org/node/10099">SimpleTest Roadmap for Drupal 7</a>.)</p>
<p>In Drupal 8, PHPUnit testing was added to Drupal Core. PHPUnit tests are much faster than a full functional test in SimpleTest—though runtest.sh still triggers a combination of these test types in Drupal 8.</p>
<p>The actual implementation of automated testing was much more complicated that this history suggests. The original testbot infrastructure that ran for 7 years on Drupal.org hardware was manually managed by some fiercely dedicated volunteers. The manual nature of that maintenance led to the architecture of DrupalCI, which was meant to make it easier to test locally at first and later focused on autoscaling on powerful hardware that could plow through tests more quickly.</p>
<h2 id="drupalcis-basic-structure">DrupalCI’s basic structure</h2>
<p>In <a href="/2016/05/18/the-drupalorg-complexity/">The Drupal.org Complexity</a>, we could see the intricate ways that Drupal’s code base interacts with other parts of the system.</p>
<p><img src="/public/images/the-drupalorg-complexity.png" alt="Representation of the relationships between services and sites in the Drupal.org ecosystem." /></p>
<p>We could further break out how systems like DrupalCI are interrelated.
<img src="/public/images/the-drupalorg-complexity-highlight-testing-integrations.png" alt="Highlighted relationships between testing and other services." /></p>
<p>DrupalCI is a combination of data stored on Drupal.org, cron jobs, drush commands, and most importantly a couple of Jenkins installations to manage all the automation.</p>
<p>Jenkins is the open source automation server project that makes most of the system possible. We use it for automating our build process and deploying code to our dev, staging and production environments. It automates just about anything and is used by companies small and large to run continuous integration or continuous deployment for their applications. It’s considered a “best practice” solution alongside options like Travis, CircleCI, and Bamboo. They all have slightly different features, but automation is at the core of most of these DevOps tools.</p>
<p>To provide continuously integrated tests, you need to trigger those tests at a moment when the tests will have the greatest value.</p>
<p>The three triggers for running a test job are when a patch is added to an issue comment, when code is committed to a repository or daily on a cron. Maintainers can specify which triggers are associated with which branches of their projects and which environments should run those tests.</p>
<p>For core these settings look something like this:</p>
<p><img src="/public/images/drupal-core-automated-testing-settings.png" alt="Screenshot of the automated testing settings for Drupal Core." /></p>
<p>This detail allows for specific tests to run at specific times per the <em><a href="https://www.drupal.org/node/2696421">Drupal.org Testing Policy for DrupalCI</a></em>.</p>
<p>To make this automation happen, we have an installation of Jenkins (Infrastructure Jenkins below) that is polled by Drupal.org once per minute with testing jobs to be queued.</p>
<p>These jobs live in a database record alongside Drupal.org.</p>
<p><img src="/public/images/DrupalCI-step-0-poll-for-jobs.png" alt="The infrastructure jenkins instance polls Drupal.org once per minute looking for new jobs to queue." /></p>
<p>Infrastructure Jenkins speaks to the CI Dispatcher (also Jenkins) where the testing queue regularly passes those jobs to available testbots. CI Dispatcher using an Amazon Web Services EC2 plugin to spin up new testbots when no existing testbot is available. Each testbot can spin up Docker containers with the specific test images defined by the maintainer. Theses containers pull from DockerHub repositories with official combinations of PHP and database engines that Drupal supports.</p>
<p><img src="/public/images/DrupalCI-step-1-ci-dispatcher.png" alt="The CI Dispatcher maintains the queue of jobs to run. When a job is ready, it uses an EC2 plugin to use an existing testbot or spin up a new bot as needed." /></p>
<p>After a testbot is running, the CI Dispatcher is in constant communication with the bots. You can even click through to the console on CI Dispatcher and watch the tests run. (It’s not very exciting—perhaps we should add sound effects to the failures—but it is very handy.)</p>
<p><img src="/public/images/DrupalCI-step-2-listen-for-results.png" alt="Once the testbot has been spun up, the CI Dispatcher listens to it for results." /></p>
<p>Once per minute, Drupal.org polls the CI Dispatcher for test status. It responds with pending, running, failed or passed. Failed and passed tests are then pulled back into Drupal.org for display using the Jenkins JSON API.</p>
<p><img src="/public/images/DrupalCI-step-3-write-status-to-drupalorg.png" alt="Drupal.org checks for test status once per minute: pending, running, failed, passed. All the results are pulled back into Drupal.org using the Jenkins' JSON API." /></p>
<p>Tests can also be run on demand at the patch, commit or branch level using the handy “add test” and “retest” links.</p>
<h2 id="why-did-we-build-this-ourselves-why-not-use-insert-testing-platform-here">Why did we build this ourselves? Why not use [insert testing platform here]</h2>
<p>Lot’s of people have asked why we don’t use TravisCI, CircleCI or some other hosted testing solution. The short answer is that most publicly available testing systems require Github authentication and integration.</p>
<p>Additionally, our testing infrastructure is powerful because of its integration with our issue queues. Read the aforementioned <em><a href="/2016/05/18/the-drupalorg-complexity/">The Drupal.org Complexity</a></em> for more information.</p>
<p>Another reason to run our own testing is scale. To get through all of the core tests for Drupal 8 in an acceptable amount of time (about 44 mins on average), we run very large testbots. These bots have 2 processors with 8 hardware cores. With hyperthreading, that means we have 32 hardware execution threads—about 88 EC2 compute units. They are not exactly super computers, but they are very performant.</p>
<p>We average nearly 18,000 test runs per month. During our peak usage we spin up as many as 25 testbot machines—though usually we cap at 15 bots to keep costs under control. This helps us plow through our testing needs during sprints at DrupalCons and large camps.</p>
<p>We have explored using an enterprise licensed version of either Github or CircleCI with our own hardware to tackle testing. That same consideration has been given to SauceLabs for front-end testing. Right now, there is not a cost savings to tackle this migration, but that does not rule it out in the future. Testing services continues to evolve.</p>
<h2 id="accelerating-drupal-8">Accelerating Drupal 8</h2>
<p>In my first months as CTO, I was told repeatedly that the most important thing for us to work on was testing for Drupal 8. In those early days as I built out the team, we were mostly focused on catching up from the Drupal 7 upgrade and tackling technical debt issues that cropped up. In DrupalCon Austin, I had members of my team learn how to maintain the testbot infrastructure so that we could take over the process of spinning up bots and dealing with spikes in demand.</p>
<p>By early 2015, we had optimized the old testbots as much as they were going to be optimized. We moved them to AWS so we could spin up faster machines and more bots, but there were features that were waiting on the new DrupalCI infrastructure that were blocking key development on Drupal 8.</p>
<p>In March of 2015, we invited all the community developers that had helped with DrupalCI to the Drupal Association offices in Portland and sprinted with them to figure out the remaining implementation needs. The next couple of months involved tweaking DrupalCI’s architecture and cutting out any nice to have features to get something launched as soon as possible.
It is no coincidence that from the time of DrupalCI’s launch until the release of Drupal 8, progress was rapidly accelerated.</p>
<p>I am immensely proud of the work of all the community members and staff that worked directly with core maintainers to unblock Drupal 8 development and make it faster. This work was critical.</p>
<p>Thank you to <a href="https://www.drupal.org/u/jthorson">jthorson</a>, <a href="https://www.drupal.org/u/ricardoamaro">ricardoamaro</a>, <a href="https://www.drupal.org/u/nick_schuch">nick_schuch</a>, <a href="https://www.drupal.org/u/dasrecht">dasrecht</a>,
<a href="https://www.drupal.org/u/basic">basic</a>, <a href="https://www.drupal.org/u/isntall">isntall</a>, <a href="https://www.drupal.org/u/drumm">drumm</a>, <a href="https://www.drupal.org/u/mikey_p">mikey_p</a>, <a href="https://www.drupal.org/u/mixologic">mixologic</a>, <a href="https://www.drupal.org/u/hestenet">hestenet</a>, <a href="https://www.drupal.org/u/chx">chx</a>, <a href="https://www.drupal.org/u/mile23">mile23</a>, <a href="https://www.drupal.org/u/alexpott">alexpott</a>, <a href="https://www.drupal.org/u/dawehner">dawehner</a>, <a href="https://www.drupal.org/u/Shyamala">Shyamala</a>, and <a href="https://www.drupal.org/u/webchick">webchick</a>. You all made DrupalCI. (And huge apologies to all those I’m undoubtedly leaving out.) Also thank you to anyone who chimed in on IRC or in the issue queues to help us track down bugs and improve the service.</p>
<h2 id="whats-next-for-testing-drupal">What’s next for testing Drupal</h2>
<p>Most of the future state of testing is outlined in the <a href="https://www.drupal.org/node/2696421">Drupal.org Testing Policy for DrupalCI</a>.</p>
<p>Key issues that we still need to solve are related to concurrent testing improvements and new test types to support. While we have PhantomJS integrated with the test runner, there are optimizations that need to happen.</p>
<p>Testing is not an endpoint. Like much of our work, it is an ongoing effort to continuously improve Drupal by providing a tool that improves how we test, what we test, and when we test.</p>
Saying goodbye2016-05-31T00:00:00+00:00http://joshuami.com/2016/05/31/saying-goodbye<p><a href="https://www.drupal.org/drupalorg/blog/saying-goodbye">Reposted from Drupal.org</a></p>
<p>When I started on the last day of March 2014, I could never have guessed at how challenging and rewarding this role would be. That is why it is incredibly hard to say that my last day with the Drupal Association will be June 10th.</p>
<p>For more details read, <em><a href="https://assoc.drupal.org/blog/megansanicki/reorganizing-for-drupal">Reorganizing for a changing Drupal</a></em>.</p>
<p>In looking at our structure with the leadership team, Holly, Matt and I saw that our roles did not fit in a smaller, more focused Drupal Association. It is no easy decision to include yourself in proposed cuts. I can’t say how proud I am to be part of a group willing to make that sort of sacrifice for something in which they believe.</p>
<h2 id="drupal-is-a-strong-community">Drupal is a strong community</h2>
<p>Drupal.org is a complex site and with a reduced engineering team, they will need to focus on mission critical services and find new ways to work with the community to continue to make the site great for contributors and evaluators.</p>
<p>With help from the rest of the team, I wrote a post called <em><a href="http://joshuami.com/2016/05/18/the-drupalorg-complexity/">The Drupal.org Complexity</a></em> that tries to take all of the complexity that surrounds our Git repositories and their integration with projects and issues on Drupal.org and describe it in a way that begins to show just how amazing our tools are. (I have yet to find a project with more total contributors to a single version of the code.)</p>
<p>Our smaller team is going to need to focus on sustaining services like testing, packaging, security updates, etc. They will be updating the <a href="https://www.drupal.org/drupalorg/roadmap">Drupal.org roadmap</a> with a more narrow focus.</p>
<p>I personally believe in the power of Drupal as an engine for amazing. And so much great stuff is still in store for the community.</p>
<h2 id="thank-you">Thank you</h2>
<p>Firstly, thanks to my family for putting up with the crazy hours, and travel that required bedtime calls at 6 in the morning—at least we had fewer pager calls in the middle of the night. Even when there were challenges, I always had your support. (And I’m looking forward to a little summer vacation together.)</p>
<p>Thank you to the board of directors for giving me this chance to help a project I love strengthen its home. Drupal.org is more professionalized and easier to maintain while continuing to scale up to the needs of the community. You took on tough discussions—and helped us make even tougher decisions. Being a board member is a huge commitment that you all took seriously while still being a blast to hang out with during and after each board meeting.</p>
<p>Thank you to my leadership teammates (Holly, Megan, Matt… and Joe too). The were some great times and some hard times. We weathered so much together with a commitment to making the best choices we could with the information we had. I can’t put into words how much I appreciated being surrounded by such a great team.</p>
<p>Thank you to my engineering staff, or tech team, or Drupal.org team—whatever we are calling ourselves this week. You’ve taught me a ton every day. I loved sharing technical challenges with you all. I’m confident this was one of the smartest teams I will ever have the chance to lead. Every single one of you have admiration. Thank you for letting me be your “umbrella” for a bit. I hope our careers cross paths again and again.</p>
<p>The entire staff at the Drupal Association is such a phenomenal group of humans! You’ve made going to work each day fun and exciting. You showed what it means to have passion and lived the values of a great community.</p>
<p>Lastly, I also want to thank the Drupal community members I was able to meet both in-person and virtually over the past two years. You are truly committed and passionate. Thank you for being patient and accepting and generous.</p>
<p>Thank you.</p>
<h2 id="whats-next">What’s next?</h2>
<p>In the next two weeks, I’ll be documenting systems to help with the transition. Specifically, the team and I will write additional posts about our testing infrastructure (DrupalCI) and our new Composer endpoints.</p>
<p>I’ll also be working very hard to help staff that are impacted by the layoffs land softly at new opportunities. If you have opportunities at your company or organization, please contact me if you are interested in some of the best Drupal talent on the market. I’m confident they won’t be available for long.</p>
<p>I hope to announce my own next steps shortly. I plan to keep involved in the community. Drupal has been a big part of my career, but it is not the only technology that has shaped it. Drupal will continue to be an important part of my toolkit for helping companies, governments, and organizations of all types delivering solutions that are complex and need to scale.</p>
The Drupal.org Complexity2016-05-18T00:00:00+00:00http://joshuami.com/2016/05/18/the-drupalorg-complexity<p><a href="https://www.drupal.org/drupalorg/blog/the-drupalorg-complexity">Cross-posted on Drupal.org</a></p>
<p>At DrupalCon New Orleans, during both <a href="https://events.drupal.org/neworleans2016/driesnote">Dries’s keynote</a> and at the <a href="https://events.drupal.org/neworleans2016/sessions/state-drupal">State of Drupal Core Conversation</a>, question of whether/when to move to Github came up again.</p>
<p>Interestingly, I was already deep into researching ways we could reduce the cost of our collaboration tools while bringing in new contributors. This post is meant to serve as a little history of how we got to where we are and to provide information about how we might choose to go forward.</p>
<h2 id="its-complex">It’s complex</h2>
<p>To say Drupal.org is complex is an understatement. There are few systems with more integration points than <a href="https://www.drupal.org/drupalorg/ecosystem">Drupal.org and its related sites and services</a>.</p>
<p><img src="/public/images/drupalorg-ecosystem.png" alt="The Drupal.org ecosystem of sites and services." /></p>
<p class="caption">The ecosystem is complex with lots of services that share integrations like login (Bakery single sign on) and cross-site code/themes.</p>
<h2 id="it-all-starts-with-the-code">It all starts with the code</h2>
<p>The slogan “come for the code stay for the community” is accurate. The community would not exist without the unifying effort of collaborating to create the code. The growth of the community is primarily because of the utility the code provides and the (relative) ease of creating a wide range of websites and applications using Drupal core combined with contributed modules and themes that allow that framework to be extended.</p>
<p>Drupal.org was an extension of the development of Drupal for a very long time. Up until Drupal 6, Drupal.org was always upgraded the day of the release of a new version. When Drupal was smaller with a more limited scope, this made a lot of sense. With the release of Drupal 6 and the surge in usage of Drupal, more and more contributors started working on the Drupal.org infrastructure and creating new sites and services to speed the collaborative work of the community.</p>
<p>One of the biggest transitions during the Drupal 6 lifecycle and community surge was The Great Git Migration. Much of the complexity of Drupal.org and the related sites and services was created during this time period. Understanding that timeline will help in understanding just how much work went into Drupal.org and Drupal at that time.</p>
<h2 id="the-great-git-migration">The <em>Great Git Migration</em></h2>
<p>In the <a href="https://groups.drupal.org/drupal-org-git-team">Great Git Migration</a>, all of the history of Drupal code was migrated to Git from CVS. The timeline for migrating to git was about what you would expect. Community conversation took time, getting volunteers to start the process took time, finally, there was a phase were dedicated (paid) resources were contracted to finish the work.</p>
<ul>
<li><strong>May 2009</strong> - Google Summer of Code project proposed to <a href="https://groups.drupal.org/node/21624">create tools to make the migration of the version control system possible</a> on Drupal.org.</li>
<li><strong>February 2010</strong> - <a href="https://groups.drupal.org/node/48818">Discussion to select the next version control system</a> was kicked off; at the time, it was a choice between Git, Baazar and Mercucial.</li>
<li><strong>July 2010</strong> - <a href="https://www.drupal.org/project/great_git_migration">The Great Git Migration project</a> is created on Drupal.org to manage the related issues and planning.</li>
<li><strong>February 2011</strong> - <a href="https://www.drupal.org/commitlog/commit/4812/624500e1565aa4cdbb12e44a69c878c0c9652169">First commit to the Git repositories</a> by the migration bot.</li>
<li><strong>July 2011</strong> - The Great Git Migration project and discussions around version control selection are last updated. This represents the semi-official end of the migration project.</li>
<li><strong>August 2011</strong> - The <a href="https://groups.drupal.org/node/313068">“let’s move to Github” wiki/discussion</a> is kicked off.</li>
</ul>
<h3 id="our-repos-are-vast">Our repos are vast</h3>
<p>We have over 35,000 projects that total over 50 GB on disk.</p>
<p><img src="/public/images/drupalorg-projects-git.png" alt="Representation of repos related to projects." /></p>
<p>All of the Git repos on Drupal.org are associated with Projects.(e.g. modules, themes, distributions, etc.)</p>
<h2 id="we-have-issues">We have issues</h2>
<p>At the end of 2015, there were nearly 900,000 issues on Drupal.org. Drupal core alone has over 74,000 issues—over 14,000 of those issues are open. Open issues is not an indicator of code quality, but it is an indicator of how many people have contributed to a project. In general, the more issues a project has, the more challenging it is for maintainers to continuously triage those bug reports, feature requests, plans, tasks and support requests.</p>
<p><img src="/public/images/drupalorg-projects-git-issues.png" alt="Representation of issues related to projects and Git repos." /></p>
<p>The issue queues are part project management, part bug tracking. As such, they are organic and messy and have lots of rules that have been documented over years of community development. We have 23 pages of documentation dedicated to explaining how to use the issue queues. There is additional documentation dedicated to how to properly fill out an issue for core, for Drupal.org, and for numerous other contributed projects.</p>
<h3 id="issues-are-integrated-into-collaboration-on-drupalorg">Issues are integrated into collaboration on Drupal.org</h3>
<p>Those issues belong to projects and are connected to the Git repos through hooks that show a system comment when an issue is related by node ID (issue number) to a commit in Git.</p>
<p>Issues can have patches uploaded to them that are the primary means of suggesting a change to code hosted on Drupal.org. The patch-based workflow has extensive documentation, but it is not a simple task for a novice user to jump in and start contributing.</p>
<p>Most Git hosting solutions (Github, Gitlab, Bitbucket, etc.) either have some version of an issue or at least integrate with an issue tracking system (Jira, Pivotal Tracker, etc.) and provide pull request functionality (a.k.a. merge requests).</p>
<p>Having the same name is where the similarities and consistencies stop. Issues on Drupal.org have status, priority, category, component, tagging and more that are unique to Drupal project workflow. It would be a significant exercise to remap all of those categorizations to a new system.</p>
<h2 id="packaging">Packaging</h2>
<p>If the projects are what you can browse and find, and the issues are how you collaborate and change the code, the next most important service for Drupal is likely the packaging system.</p>
<p><img src="/public/images/drupalorg-projects-packaging.png" alt="Representation of projects and repos related to packaging." /></p>
<p>Packaging is based on project maintainers creating a <em>release</em> of the code by associating a branch of the Git repository with the release. Every 5 minutes, our automation infrastructure checks for new releases and will package those releases into a downloadable file to represent the project.</p>
<p>Few developers actually access this directly from the project page anymore. They are much more likely to use Git, Drush, Console or Composer to automate that part of the workflow. Drush, and to some extent Composer, both use the packaged files when a command is issued. Also, the Drupal feature of just putting the code in the correct directory and it will run—with no compiling—is fundamental to the history of Drupal site building.</p>
<h2 id="updates">Updates</h2>
<p><img src="/public/images/drupalorg-projects-updates.png" alt="Representation of projects and repos related to updates." /></p>
<p>Another crucial Drupal service, updates is built into how Drupal core checks on itself to see if it is up to date.</p>
<p>The 1.3 million plus websites that call home to updates.drupal.org get back XML that is then parsed by that installation’s update status module; that updates module has different names depending on the version of Drupal. Each month, about 12 terabytes of traffic to our CDN is requests for updates XML. Considering this is a bunch of text files, this is an amazing number to consider. Some sites call home once a week, some once a day, and some do it every few minutes. (Really people! Be nice to your free updates service. Telling your server to ask for updates daily is plenty frequent enough.)</p>
<p>Tallying the unique site keys that request this information is how we get our <a href="https://www.drupal.org/project/usage">usage statistics</a>. While this is probably not the most precise way to measure our usage, it is directionally accurate. There are plenty of development sites in those stats and plenty of production websites that don’t call home. It should roughly balance out. To be anymore precise, users of Drupal would have to give up some privacy. We’ve balanced accuracy with privacy.</p>
<p>Because of our awesome CDN (thanks, Fastly!), we are able to deliver up to date packages and updates information in milliseconds after we update the underlying data.</p>
<h2 id="composer">Composer</h2>
<p><img src="/public/images/drupalorg-projects-composer.png" alt="Representation of projects and repos related to Composer." /></p>
<p>On May 3rd, we launched the <a href="https://www.drupal.org/node/2713789">alpha version of our Composer endpoints</a> on Drupal.org. If you don’t know about <a href="https://getcomposer.org/">Composer</a>, you should <a href="https://getcomposer.org/doc/00-intro.md">read up on it</a>. Composer is package management for PHP. (It’s similar to what NPM does for Node.js or RubyGems does for Ruby.)</p>
<p>Core developers have been using Composer for some time as a means to manage the dependencies of PHP libraries that are now included in core.</p>
<p>The <a href="https://www.drupal.org/node/2576285">Composer endpoints</a> allow any Drupal site developer to use <code class="language-plaintext highlighter-rouge">composer install</code> to build out their websites.</p>
<p>The new Composer service will also allow contrib project maintainers to using composer.json files to define the requirements for their modules and themes. The service even translates project release versions into semantic versioning. Semantic versioning was the biggest reason we could not “just” use <a href="http://packagist.org">Packagist.org</a> like other projects in the PHP community.</p>
<p>This is all a huge benefit, but more importantly, we now have deep integration between a best practice approach to PHP dependency management and the Drupal.org code repos that can scale to our community needs.</p>
<h2 id="testing-with-drupalci">Testing with DrupalCI</h2>
<p>Speaking of needs, DrupalCI ran 67,000 test runs in January 2016. Each test run for Drupal core has 18,511 tests per run. That means over 100,000 assertions (steps) in the unit and functional tests that make sure Drupal’s code is stable and that an accepted patch does not create a regression.</p>
<p><img src="/public/images/drupalorg-projects-issues-drupalci.png" alt="Representation of projects, repos and issues related to DrupalCI." /></p>
<p>At the time of this post, we are using Amazon Web Services cc2.8xlarge EC2 spot instances for our testbots. These bots are powerful. They have 2 processors with 8 hardware cores. AWS claims they can provide 88 EC2 compute units per instance. They are packed with processing power because we have a lot of tests to run. While there are bigger instances, the combination of price and power allows us to keep Drupal core complete test runs right around 30 minutes. We autoscale up to 20 of these instances depending on demand, which keeps queue times low and allows maintainers to get quick feedback about whether a patch works or not.</p>
<p>I truly believe that getting DrupalCI up and stable is what allowed Drupal 8 to get to a full release last fall. Without it, we would have continued to struggle with test times that were well over an hour and a system that required surplus testbots to be manually spun up when a big sprint was happening. That was costly and a huge time waste.</p>
<p>If anyone asks me “what’s the most important thing your team did in 2015”, I can unequivocally say “unblocking core development to get Drupal 8 released.”</p>
<h2 id="issue-credits">Issue credits</h2>
<p>The second most important service we built in 2015—but certainly the more visible—is a system for crediting users and organizations that contribute on Drupal.org.</p>
<p><img src="/public/images/drupalorg-projects-issues-credits.png" alt="Representation of projects and issues related to credits." /></p>
<p>Issue credits sprang forth from an <a href="http://buytaert.net/a-method-for-giving-credit-to-organizations-that-contribute-code-to-open-source">idea that Dries proposed</a> around DrupalCon Austin in June of 2014. At the time, his intent was a means of structuring commit messages or using commit notes to provide the credit. Eventually, we shifted the implementation to focus on participation in issues rather than code commits. This made it possible to credit collaboration that did not result in a code change.</p>
<p>I won’t get into the specifics; I wrote a <a href="http://joshuami.com/2016/01/11/guide-to-issue-credits-and-marketplace/"><em>A guide to issue credits and the Drupal.org marketplace</em></a> earlier this year. Issue credits have been extremely successful.</p>
<p>As there name implies, we store the data about credits as a relationship to closed issues. Issue credits touch issues, projects, users, organizations and the marketplace on Drupal.org.</p>
<h2 id="why-not-just-migrate-all-of-this-complexity-to-github">Why not just migrate all of this complexity to Github?</h2>
<blockquote>
<p><em>Why can’t we just move all this to Github?</em><br />
— said <a href="https://groups.drupal.org/node/313068">lots of people</a>, often</p>
</blockquote>
<p>To be fair, this is a challenging discussion. Angie Byron (<a href="https://www.drupal.org/u/webchick">webchick</a>) wrote an amazingly concise <a href="https://groups.drupal.org/node/313068#comment-955593">summary of the Github issue</a> on Groups.drupal.org.</p>
<p>That wiki/discussion/bikeshed was heated. The conversation lasted over two years. I started as CTO about 6 months into the conversation. Along with a couple of other themes, the Github move has been a constant background conversation that has defined much of my time leading the Drupal.org team.</p>
<h2 id="how-are-these-services-connected">How are these services connected?</h2>
<p>To truly understand the problem of a migration of this scale, we have to look at how all of the major Drupal.org services are connected.</p>
<p><img src="/public/images/the-drupalorg-complexity.png" alt="Representation of the relationships between services and sites in the Drupal.org ecosystem." /></p>
<p>Each block in this diagram is a service. Each line is a point of integration between the services. Some of these <em>services</em> are on Drupal.org or subsites with thousands of lines of custom code defining the interactions. Other <em>services</em> are not built in Drupal and represent projects in Java (Jenkins) or Python (our Git daemon) with varying degrees of customization and configuration.</p>
<p>As the diagram suggests, it is truly a <em>web</em> of integrations. Pull one or more services out of this ecosystem and you have to either refactor a ton of code or remove a critical component of how the community collaborates and how our users build sites with Drupal.</p>
<p>It’s kinda like a great big game of <a href="https://en.wikipedia.org/wiki/Jenga">Jenga</a>.</p>
<h2 id="what-would-a-migration-to-github-require">What would a migration to Github require?</h2>
<p>Please believe me when I say that if it were “easy” or “simple”, we would have made either moved to Github or at least <a href="https://www.drupal.org/node/2488266">upgraded our Git collaboration with nifty new tools</a> on our own infrastructure.</p>
<p>However, disrupting the development of Drupal 8 would have been devastating to the project. We were correct to collectively backlog this project.</p>
<p>So if we were to try this migration now, what would it take? First, you have to consider the services that Github would effectively replace.</p>
<p>Github replaces:</p>
<ul>
<li>Git repositories</li>
<li>Issues</li>
<li>Patches (they would become pull requests)</li>
<li>Git viewing (and we’d get inline editing for quick fix and onboarding)</li>
</ul>
<p><img src="/public/images/the-drupalorg-complexity-highlight-github-impact.png" alt="Areas of the Drupal.org system integration map with services that Github could replace highlighted (4) and integrations points that would need to be refactored (16)." /></p>
<p>That’s four (4!) services that we would not have to maintain anymore. Awesome! Cost savings everywhere! Buy a boat!</p>
<p>Wait a second. You have 16 integration points that you need to refactor. Some of them would come with the new system. Issues, pull requests, repos and the viewer would all just work with huge improvements. That leaves us with 12 integration points that would require a ton of discovery and refactoring.</p>
<ol>
<li><strong>Users</strong> - we have 100,000 Drupal.org users that are pretty engaged. (We have over 1 million user accounts—but that number is likely a little inflated by spam accounts.) Do we make them all get Github accounts? Do we integrate Github login to Drupal.org? Do we just link the accounts like Symfony does?</li>
<li><strong>Projects</strong> - Github is not a project browsing experience. Drupal.org is a canonical repository where the “one true project” lives for packaging and updates. At the very least, we have to integrate our projects with Github. Does that mean we have to keep a Git repo associated to the project that has hooks to pull in changes from Github?</li>
<li><strong>Testing</strong> - One of the less complex integration refactors would be getting DrupalCI integrated with pull requests. That effort would still be a months long project.</li>
</ol>
<p>And DrupalCI would be its own effort to migrate to another testing service because it is tailored to the issue queue workflow and tightly integrated with projects.</p>
<p><img src="/public/images/the-drupalorg-complexity-highlight-testing-integrations.png" alt="Areas of the Drupal.org system integration map with the testing service (DrupalCI) highlighted and the integration points that would need to be refactored (4)." /></p>
<p>Those are just a few of the major integration points.</p>
<p>I have a personal goal to detail every single integration and get that documented somewhere on Drupal.org. I don’t think that level of documentation will increase the ability for others to contribute to the Drupal.org infrastructure—though that would be a pleasant side effect. I do think it is necessary for us to continue to support and maintain our systems and ensure that all of the tribal knowledge from the Drupal.org team can be passed on.</p>
<h2 id="what-would-it-cost">What would it cost?</h2>
<p>I have joked that it would take roughly 1 million dollars (USD) to complete a Github migration. (<a href="https://www.youtube.com/watch?v=l91ISfcuzDw">Cue Dr. Evil.</a>) That is only partially meant in jest.</p>
<p>As anyone who has estimated a large project knows, there is a point of uncertainty that leads project owners to guess at what they are <em>willing</em> to pay for the project.</p>
<p>If we take the four biggest lifts in the Drupal project’s history, what do we get?</p>
<ol>
<li><strong>Drupal.org redesign</strong> - There were tens of people involved in the project, hundreds giving feedback. The timeline was about a year from start to implementation.</li>
<li><strong>The Great Git Migration</strong> - There were tens of people involved in the project. Far fewer users gave feedback, but the project took about two years from brainstorming to initial commit to the Git repos—with a few months of clean up after.</li>
<li><strong>Drupal.org upgrades to Drupal 7</strong> - The project took about two years with tens of people involved in about 8 months of clean up issues.</li>
<li><strong>Drupal 8</strong> - 5 years of development by over 3,000 contributors.</li>
</ol>
<p>I don’t think than anyone would argue that each of these projects would have been bid at well over $1 million. I would put a migration to Github at somewhere between the complexity of The Great Git Migration and Drupal 8.</p>
<p>In none of these cases did the Drupal Association actually spend $1 million USD in project dollars. However, in all of the projects, there was lengthy discussion followed by substantial volunteer contribution, and then a significant bit of paid work to finish the job. It’s a pattern that makes sense and will likely repeat itself over and over.</p>
<h2 id="would-it-be-worth-it">Would it be worth it?</h2>
<p>I’m going to go back to the <a href="https://groups.drupal.org/node/313068#comment-955593">summary on the Github discussion</a>. There are reasons why both options seem to be the best possible option or the worst possible option.</p>
<p>Would a best practice workflow and toolset be worth the change? Absolutely. Github (or Gitlab) tools are easier for newcomers to learn. Further, because we are using PHP and Javascript libraries that are hosted on Github, we could get contributions from developers and designers that are involved in those projects and do not wish to have account on Drupal.org.</p>
<p>The drawbacks are considerable. We cannot afford a full migration right now. Dries put it well at DrupalCon Los Angeles during core conversations. The Drupal Association is not a bag of money. With significant growth of revenue, there is a long term possibility of more paid developer resources, but not in the short term. It is too much to ask volunteers to give up a year of their life to run the project as a community initiative. That leads to burn out and frustration.</p>
<p>We should also consider whether the disruption to the current collaboration workflow will be worth it. I don’t think so. Not if that disruption meant stalling the update of contrib projects that are critical to solidifying Drupal 8 adoption. (Though I could argue that much of this upgrade to Drupal 8 work is being performed on Github as some—perhaps many—developers prefer those tools.)</p>
<h2 id="is-there-a-middle-ground">Is there a middle ground?</h2>
<p>Drupal spends a lot of time getting to the middle ground. Many of the best innovations in Drupal come from getting to the middle ground—from reaching a general consensus and then allowing someone who has support and time iron out the details.</p>
<p>So for the first step, we <strong>should</strong> add functionality to projects on Drupal.org that allow maintainers to shift their workflow to Github while still publishing their project on Drupal.org. This allows the canonical browsing of projects, the continued support of the security team, and most importantly the continued distribution of Drupal through Composer, release packaging and the updates system.</p>
<p>We have a solid way forward for these integrations as the requirements are narrow enough in scope to accomplish in a 4-6 month timeframe using dedicated resources. We would still need to figure out how to award an issue credit to someone that participated in an issue on Github. We might be able to institute commit credits that could be parsed into issue credits from the participation on Github, but it would not be as inclusive as the current model.</p>
<p>It would be important to phase in this new feature rather than make a wholesale change. Once that integration is in place, we could extend DrupalCI to test pull requests similar to how we currently test patches submitted to an issue.</p>
<h2 id="stay-flexible">Stay flexible</h2>
<p>We need to be flexible. GitHub has a lot of potential as tool for open source distribution and collaboration—likely for the foreseeable future. However, not every major project is on GitHub. The Linux Kernel uses Git repositories with cli tools and a patch-based workflow that relies heavily on email. It works for them. Wordpress is still on Subversion—even though they’ve started to accept some pull requests on GitHub. These projects are poised to make the right decision rather than a rash decision.</p>
<p>The sky will not fall if we keep our current model, but we are losing opportunities to grow as a community of contributors. Rather than a wholesale migration, we must understand the value and history of this web of integration points. Targeting our efforts on specific integrations points can achieve our goal of opening our doors to the developers who live and breathe GitHub, without losing the character of our collaboration. And in the long run, this focus on services and integrations can make us more adaptable to the next change in the broader development landscape.</p>
<p>Special thanks to Tim Lehnen, Ryan Aslett, Rudy Grigar and Neil Drumm for helping me pull this data together and to the rest of the Drupal.org team for reviewing the post for typos.</p>
Principles of technology that can improve governments at every level2016-04-07T00:00:00+00:00http://joshuami.com/2016/04/07/improve-every-level-of-government-through-technology<p>Improving government through technology will require change in how we think about services provided at the city, local, state, and federal levels.</p>
<p>The following 6 principles are key to using technology to enable better government services:</p>
<ol>
<li>Build for change</li>
<li>Services are for people</li>
<li>Procure wisely</li>
<li>Transparency by default</li>
<li>Make decisions with data</li>
<li>Legislate for simplicity</li>
</ol>
<h2 id="build-for-change">Build for change</h2>
<p>Technology will change. Our workforce will change. Our cities and services will change.</p>
<p>We are unlikely to move our governments from a yearly budget to a rolling forecast and budget that changes quarterly. However, we can take steps to make sure that our systems and people are better prepared for constant change.</p>
<p>Iterative design and development is a must to allow technology systems to keep up with our complex service needs.</p>
<p>Projects should be planned at the scale of one year or less. All projects that take more than one year to initially implement will fail. Positions should be hired with a focus on flexibility and mobility. Permanent headcount is fine, but we should never assume that a position will exist forever based on technology and services that constantly change.</p>
<h2 id="services-are-for-people">Services are for people</h2>
<p>A focus on user-centered design is a must. Citizens should be able to engage with services. The government agencies that produce services should have embedded product managers, designers, and developers that are helping them connect with the users.</p>
<p>User centered design will have the added benefit of creating services that cost less and are optimized for the day-to-day use of both service providers and recipients.</p>
<h2 id="procure-wisely">Procure wisely</h2>
<p>Procurement policies in government must shift to a model that recognizes the value of commodity services at the core level, then an open source development of web services. We must foster a “proprietary-last” mentality when it comes to potential vendor lock-in.</p>
<p>Our procurement policies should focus on foundational technologies rather than COTS applications that are sold as best practice, but seldom have widespread use.</p>
<p>By focusing on a open source first approach, governments can ensure that the value built for their citizens can be shared with and improved by other government agencies with similar needs.</p>
<p>Procurement of this type will be less focused on reducing large risk and more focused on producing a high return on investment.</p>
<h2 id="transparency-by-default">Transparency by default</h2>
<p>Data should be open by default with a strong bias to protecting personal information. At first, these two goals may sound like they are in opposition. When we begin any new data collection, we should first think about how that data will be sanitized and depersonalized to allow it to be shared.</p>
<p>By making our open data available as public web services, we will be allowing private industry and opportunity to innovate with information that was previously unavailable.</p>
<p>Transparency relates to software development as well. Using open source and contributing back results in less hidden costs in technology and better security through public review.</p>
<p>Transparent practices and open data extends the reach of government services to the benefit of everyone.</p>
<h2 id="make-decisions-with-data">Make decisions with data</h2>
<p>Data is the key to effective systems. We must ensure decision makers can access it and act on it.</p>
<p>Any decision requested of our elected officials or administrative leaders should be accompanied by data. Requests that are not supported with vetted data or that have explicit unsupported assumptions should be suspect.</p>
<h2 id="legislate-for-simplicity">Legislate for simplicity</h2>
<p>In order to actively improve government through technology, we must have technologically literate elected officials that are willing to change laws to simplify complex systems.</p>
<p>Unchecked complexity can lead technology projects to failure. Loopholes and exceptions are difficult to program into software. Software that is complex costs more to maintain and change.</p>
<p>In order to be good stewards of our resources, we must choose the simplest, most cost effective services that our government is prepared to deliver. This requires a high level of systems thinking and a willingness to do less in order to expedite the delivery of those services.</p>
<h2 id="conclusion">Conclusion</h2>
<p>By following these principles, and constantly evaluating our approaches looking for improvement, we can make government services more effective. We can build and buy great services for our citizens that can simply and cost effectively connect them to the information and services they need. We can do this important work with an organization that is optimized to continually change and willing to make the difficult policy and legislation changes necessary.</p>
<h2 id="resources">Resources</h2>
<ul>
<li><a href="https://sourcecode.cio.gov/">U.S. Office of the CIO Source Code Policy</a></li>
<li><a href="https://backchannel.com/the-tiny-team-taking-on-a-massive-reform-of-government-it-b5f87b85e2dc#.i9r4vuy1d">Backchannel: The tiny team taking on a massive reform of government</a></li>
<li><a href="https://backchannel.com/the-white-house-s-alpha-geek-20b338738929#.w14gppq8r">Backchannel: The White House’s Alpha Geeks</a></li>
<li><a href="http://www.codeforamerica.org/practices/">Code for America: 21st Century Government Practices</a></li>
<li><a href="http://www.aspeninstitute.org/publications/city-platform-how-digital-networks-are-changing-urban-life-governance">The City as PlatForm: how digital Networks are Changing
Urban life and governance, David Bollier, The Aspen Institute</a></li>
<li><a href="https://18f.gsa.gov/2015/01/08/creating-a-federal-marketplace-for-agile-delivery-services/">Creating a Federal Marketplace for Agile Delivery</a></li>
</ul>
My introduction to FIWARE2016-02-18T00:00:00+00:00http://joshuami.com/2016/02/18/my-introduction-to-fiware<p><a href="https://medium.com/@joshuami/my-introduction-to-fiware-d8991bcbbb3c#.1wqo4hfm3">Republished on Medium</a></p>
<p>Among <a href="http://joshuami.com/about">my interests and volunteer activities</a>, I sit on the City of Portland Technology Oversight Committee. My role in that committee is to give feedback to City of Portland IT leadership on large-scale, high-visibility technology projects. This can range from projects to revamp water billing to systems managing loans and liens in the housing bureau to the attempted rewrite of all of the City’s recording and permitting systems. When something goes wrong on one of these big projects, usually a local paper will write about it. Through monthly updates and quarterly public council reports, the TOC gives the City council a dashboard of how these projects are doing based on independent review of major project deliverables. I won’t lie, sometimes these reports are boring as hell… but usually there is something about these projects that make them a wee bit exciting to a tech and government geek like me.</p>
<p>I serve on this committee with some great people that are truly committed to making technology in government work. It is nice to know there are others dedicated enough to give an evening a month—and a few hours of prep besides—to make sure the City is being good stewards of our tax dollars.</p>
<p>When one of my fellow members, Wilf Pinfold, invited me to an event to meet members of a FIWARE delegation at the Jaguar Land Rover Accelerator here in Portland, I jumped at the chance. (And yes, they have nice offices… but much less leather and wood paneling than you might expect.) <a href="https://www.fiware.org/">FIWARE</a> “is an independent open community whose members are committed to materialise the FIWARE mission, that is: ‘to build an open sustainable ecosystem around public, royalty-free and implementation-driven software platform standards that will ease the development of new Smart Applications in multiple sectors’.” The members of the delegation were connected to Wilf, and his company urban.systems Inc., through relationship with <a href="http://www.nist.gov/">NIST</a> (the National Institute of Standards and Technology) and their Smart Cities initiative.</p>
<p>I’ve been a huge proponent for open source and open data in government for years. My role as CTO for the Drupal Association has meant that I’ve had ample opportunity to meet project leads and influencers from a lot of amazing projects, but I had not come across FIWARE yet. After the introduction, I understand why.</p>
<p>FIWARE has been predominantly a European movement. Of the 75 cities that have signed on to support FIWARE, most of those cities are in Europe. (The notable exception is in Mexico.) The movement has only gained significant traffic in the last year. There are more cities in the Western hemisphere poised to sign on, but there is still some question as to whether this will pick up in the U.S.</p>
<p>So what did I think? Well, that’s complicated. I like the idea in principle. Heck, this is what I love about open source. You get a lot of people together in shared economy and drive innovation that benefits everyone. Government should get better. New companies should pop up all around the governments involved to take advantage of new data unleashed on the marketplace.</p>
<p>As for the pitch from the delegation, they need to focus a bit more on the real use cases and less on the marketing materials around their community. To truly grow an open source project quickly, you need only show that it works and delight the developers with new skills that are challenging, rewarding, and ideally give them a career path. Javascript has exploded in the open source world in the past year not because of the promise of one platform to run on both the server and the browser, but because the stuff being made with Angular, Ember, React and more are frankly fun to build. They are new and interesting challenges.</p>
<p>Arguably, the FIWARE stack is not that sexy. It is built out on the principle of solid stack management (Openstack) combined with reusable “enablers” (mostly Python and Java projects combined with predefined containers—usually Docker) that make it <em>easier</em> to snap together an application. (Emphasis is to suggest that is a relative term.) Some of these look like good solid solutions, but they are certainly not best of breed in all cases. That said, I like how they are getting companies to contribute this code back with clear licensing—mostly MIT, Apache2, and GPLv3. I even learned about a license I had not heard of before—the EUPL (European Union Public License). (I have no idea what license compatibility looks like for the EUPL, but I am planning to research it a bit.)</p>
<p>Some of these technologies were clearly added to the catalog in 2009-2012 as the initial FIWARE collaborators where pulling together resources. As a result, much of the catalog is a bit dated-but-maintained. (I can live with that. Drupal seems a bit dated-but-maintained in a few areas.) I couldn’t help but spend large portions of the meeting comparing features with what I have seen built with Drupal and other open source projects. Drupal fares extremely well in the government space. There are great shops doing work to get out public data and to publish amazing public service directories. (My time at Multnomah County showed that you can build a lot of government value quickly using a combination of open source and SaaS solutions.)</p>
<p>While the technology in the catalog is not exciting, the demonstrations of the products in action were a bit more impressive.</p>
<p>In Spain, a company that started in a FIWARE incubator is working on an application to allow commuters to pick a destination and get a list of ride share, car share, bike share, and transit options and pick the option that most fits their desired budget, speed, or health concerns. (Biking there will always win the healthy option—but I digress.)</p>
<p>In Portugal, Porto (the Portland of Portugal according to the NYTimes) is collaborating with a local university to blanket the city in environmental sensors that measure air quality, noise levels, temperature and more to help improve livability and give government decision makers better data.</p>
<p>Those were both great uses of the FIWARE platform, but they relied on components (services and data) outside of the platforms.</p>
<p>So the foundational software leaves a little to be desired, but it is a good start. It serves the purpose of speeding time to market for products needed in smart cities, sustainable transport, and logistics.</p>
<p>The commitment to getting demonstrations of product into the hands of government influencers is commendable and an excellent start. I would like to see FIWARE reach out to open source projects with experience in government to tie in these solutions. Drupal would be a great place to start with our significant number of installations at all levels of government throughout the world.</p>
<p>Another interest that I have in FIWARE is how they are funded. In contrast to the Drupal Association which, as a non-profit, is principally funded by the conferences (DrupalCons) we put on each year and our supporting partner program, is funded mostly by the people that benefit from Drupal. We get very little funding from those that are customers of Drupal the software. FIWARE is funded through a public-private partnership. They have accelerators and will fund entrepreneurs up to €150,000 to build a solution using FIWARE. The practicality of having a government resource that drives the development of open source and open data is a huge velocity builder for the community. I would love to figure out how to repeat this model with other open source projects. Especially projects used so heavily in government already such as Drupal.</p>
<p>FIWARE itself is in the midst of a transition from a public-private partnership with organizations committed to releasing code under a shared interest to a full-fledged open source community. It sounds like they are planning to launch a foundation to fund and support ongoing efforts.</p>
<p>After my introduction to FIWARE, I left with three basic thoughts.</p>
<ul>
<li>FIWARE has slick marketing materials for standing in front of decision makers.</li>
<li>The software solutions often are not as advanced as the latest from popular open source projects.</li>
<li>Funding matters. To build out ecosystems like FIWARE takes a ton of commitment by business, government, and passionate developers that want to see smarter technologies put in place that improve everyone’s quality of life.</li>
</ul>
<p>I’m excited see what happens next with FIWARE. They have huge barriers to overcome, but their basic principle of using open source to make government better is something I can get behind.</p>
A (fairly) extensive list of contribution types on Drupal.org2016-01-12T00:00:00+00:00http://joshuami.com/2016/01/12/comprehensive-list-of-contribution-types-drupalorg<p><em><a href="https://www.drupal.org/node/2649100">Contribute to this big long list on Drupal.org</a></em></p>
<p>This is an extensive list of contribution types that we either track or could track. It should be noted that this is not comprehensive. There are forms of contribution that do not leave behind data that cannot be collected. That said, this list does cover lots of types of contribution and not just code.</p>
<p>If we were to apply some math to this data, we could calculate an engagement score for users and organizations. Tracking engagement over time could be a key health metric.</p>
<p>Many of these contribution types would make profiles and organization pages more robust and would be less subjective than self-reported information.</p>
<table>
<thead>
<tr>
<th style="text-align:left">Contribution Type</th>
<th style="text-align:center">User</th>
<th style="text-align:center">Org</th>
<th style="text-align:left">Status</th>
<th style="text-align:left">Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">Issue credits</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Displayed on profile</td>
<td style="text-align:left">Only last 90 days are displayed, but all are tracked. Awarded by project maintainer</td>
</tr>
<tr>
<td style="text-align:left">Commits</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Displayed on profile</td>
<td style="text-align:left">Git records</td>
</tr>
<tr>
<td style="text-align:left">Projects owned</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">By author field (can change and could be broken down by full versus sandbox)</td>
</tr>
<tr>
<td style="text-align:left">Projects supported</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Displayed on profile</td>
<td style="text-align:left">Ref on org profile</td>
</tr>
<tr>
<td style="text-align:left">Change notices (project participation)</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">Technically, we might show this as a post that the user created</td>
</tr>
<tr>
<td style="text-align:left">Projects maintained</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">We would need to decide how many maintainer types/roles to display</td>
</tr>
<tr>
<td style="text-align:left">Documentation revisions</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Displayed on profile</td>
<td style="text-align:left">We show this listed as an aggregate. (i.e. "more than 100 document edits") Cannot credit to orgs</td>
</tr>
<tr>
<td style="text-align:left">Documentation section maintained</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not yet tracked</td>
<td style="text-align:left">This will be possible with new documentation tools defined in the content strategy.</td>
</tr>
<tr>
<td style="text-align:left">Forum posts created</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">Technically, you can see these as posts by the user, but it is not a total and not limited to forum posts.</td>
</tr>
<tr>
<td style="text-align:left">Forum comments</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">Technically, you can see these as comments by the user, but it is not a total and not limited to forum comments.</td>
</tr>
<tr>
<td style="text-align:left">DrupalCons sponsored</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">We have this for most past DrupalCons.</td>
</tr>
<tr>
<td style="text-align:left">Supporting partner status</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Displayed on profile</td>
<td style="text-align:left">Supporting partners fund Drupal.org—both infrastructure and staffing costs.</td>
</tr>
<tr>
<td style="text-align:left">Membership status</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Displayed on profile</td>
<td style="text-align:left">Individual or organization membership</td>
</tr>
<tr>
<td style="text-align:left">Donations and support to the Drupal Association</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">We have totals of money spent with the Drupal Association, but we'd need to aggregate this information and likely abstract it to levels.</td>
</tr>
<tr>
<td style="text-align:left">Jobs posted</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">Technically, we could show this for users, but I'm not sure there is value.</td>
</tr>
<tr>
<td style="text-align:left">Issues created</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Displayed</td>
<td style="text-align:left">Kinda. This is really "posts" of all types on the user profile.</td>
</tr>
<tr>
<td style="text-align:left">Issue comments</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Displayed</td>
<td style="text-align:left">Kinda. This is really "comments" of all types on the user profile.</td>
</tr>
<tr>
<td style="text-align:left">Issues followed</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">This seems like an excellent engagement metric.</td>
</tr>
<tr>
<td style="text-align:left">Projects followed</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">Also an excellent engagement metric.</td>
</tr>
<tr>
<td style="text-align:left">User/interest groups owned/maintained</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">From Groups.drupal.org data</td>
</tr>
<tr>
<td style="text-align:left">User/interest group membership</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">From Groups.drupal.org data</td>
</tr>
<tr>
<td style="text-align:left">User/interest group participation</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">No displayed</td>
<td style="text-align:left">By count of nodes created or comments posted</td>
</tr>
<tr>
<td style="text-align:left">Local user groups sponsored</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">Great organization metric</td>
</tr>
<tr>
<td style="text-align:left">Camps sponsored</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">Camps would need to help us track this and submit their data to staff for upload.</td>
</tr>
<tr>
<td style="text-align:left">DrupalCon session submissions</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">We have this data for quite a few past DrupalCons.</td>
</tr>
<tr>
<td style="text-align:left">DrupalCon speaker</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">We have this data for quite a few past DrupalCons.</td>
</tr>
<tr>
<td style="text-align:left">DrupalCon Trainer</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">We have this data for quite a few past DrupalCons.</td>
</tr>
<tr>
<td style="text-align:left">BoF sessions submitted</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">We have this data for quite a few past DrupalCons.</td>
</tr>
<tr>
<td style="text-align:left">Translations strings submitted</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">From localize.drupal.org. It would be great to figure out how to attribute this work back to organizations for organization credit.</td>
</tr>
<tr>
<td style="text-align:left">Translations strings accepted</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">From localize.drupal.org. It would be great to figure out how to attribute this work back to organizations for organization credit.</td>
</tr>
<tr>
<td style="text-align:left">Translations strings reviewed</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">From localize.drupal.org. It would be great to figure out how to attribute this work back to organizations for organization credit.</td>
</tr>
<tr>
<td style="text-align:left">Core patches submitted</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">This could potentially be pull requests or merge requests with new tools</td>
</tr>
<tr>
<td style="text-align:left">Contrib patches submitted</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">This could be pull requests or merge requests with new tools</td>
</tr>
<tr>
<td style="text-align:left">Case studies submitted</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Displayed</td>
<td style="text-align:left">Partially displayed. This shows up on the marketplace listings, but not on the profile.</td>
</tr>
<tr>
<td style="text-align:left">Employees on Drupal.org</td>
<td style="text-align:center"></td>
<td style="text-align:center">X</td>
<td style="text-align:left">Displayed on profile</td>
<td style="text-align:left">This display could use some work. Acquia's page takes about 10 seconds to load because the list of users is so long.</td>
</tr>
<tr>
<td style="text-align:left">Drupal Planet Posts</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">This could be tied back to users with a little customization</td>
</tr>
<tr>
<td style="text-align:left">Camps organized</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">We might be able to pull this from the groups data with some modifications to the roles/management types.</td>
</tr>
<tr>
<td style="text-align:left">User/interest groups organized</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">We might be able to pull this from the groups data going forward</td>
</tr>
<tr>
<td style="text-align:left"># of mentees listing as a mentor</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">This is similar to the concept of "followers", but people community focused.</td>
</tr>
<tr>
<td style="text-align:left">Is board member</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">We could put this on profiles</td>
</tr>
<tr>
<td style="text-align:left">Core maintainer</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">Orgs it could be "hires a core maintain"</td>
</tr>
<tr>
<td style="text-align:left">Sprint mentor</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">Wouldn't this be awesome? We'd have to decide what sprints qualify—just DrupalCons or all big sprints at camps and Dev Days.</td>
</tr>
<tr>
<td style="text-align:left">Staff</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">Hey, we know all our staff, but most people don't know if they are talking to a full time Drupal Association employee.</td>
</tr>
<tr>
<td style="text-align:left">Elevated role on Drupal.org</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">We've given special privileges to these people. It would be nice to highlight that.</td>
</tr>
<tr>
<td style="text-align:left">Working group member</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">This would help people understand governance roles a bit better.</td>
</tr>
<tr>
<td style="text-align:left">Security team member</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">We list all of these users, but it would be nice to show it as a badge on their profile pic for interactions in the issue queues.</td>
</tr>
<tr>
<td style="text-align:left">DrupalCon volunteer</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">We have the data in spreadsheets and thank you notes. It really should show up on profiles.</td>
</tr>
<tr>
<td style="text-align:left">DrupalCon attendee</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not displayed</td>
<td style="text-align:left">Technically, people self-report this information right now, which means it is not accurate. Org would be a little difficult to track on this one, but it could possibly be done</td>
<td></td>
</tr>
<tr>
<td style="text-align:left">Global Training Days organizer</td>
<td style="text-align:center">X</td>
<td style="text-align:center">X</td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">We'd have to import this data.</td>
</tr>
<tr>
<td style="text-align:left">IRC support</td>
<td style="text-align:center">X</td>
<td style="text-align:center"></td>
<td style="text-align:left">Not tracked</td>
<td style="text-align:left">This would require people to give credit to people with IRC handles listed on Drupal.org. Maybe we could use the Karma points from IRC.</td>
</tr>
</tbody>
</table>
A guide to issue credits and the Drupal.org marketplace2016-01-11T00:00:00+00:00http://joshuami.com/2016/01/11/guide-to-issue-credits-and-marketplace<p><em><a href="https://www.drupal.org/news/guide-to-issue-credits-and-marketplace">Republished from Drupal.org</a></em></p>
<p>There’s been a lot of positive feedback and a few questions about the recent changes to the marketplace. I want to take a minute and talk about how the marketplace made it to where it is now and mention some plans for the future of the marketplace on Drupal.org.</p>
<h2>TL;DR</h2>
<p>This is a long post. You may not way to read the whole thing, so here is a condensed version.</p>
<p>We want to highlight individuals and organizations that are actively contributing to Drupal. You can now attribute your work in the Drupal.org issue queues to your employer or a customer. Maintainers can award issues credits to people who help resolve issues with code, comments, design, and more. These credits can roll up to service providers listed in the Drupal Marketplace. (And eventually we want to show customers with credits as well.)</p>
<p>Want to know all the details? Read on.</p>
<h2>A little history</h2>
<p>Back at DrupalCon Austin (2014), Dries presented a sketch of an idea for highlighting organizational contributions to Drupal to myself and several business owners in the Drupal community. Later that week, those initial ideas turned into <a href="http://buytaert.net/a-method-for-giving-credit-to-organizations-that-contribute-code-to-open-source" rel="nofollow">A method for giving credit to organizations that contribute code to Open Source</a>. </p>
<p>The idea was to use commit messages to highlight organizational contribution. The first draft looked something like this:</p>
<p><code>$ git commit -am "Issue #n by INDIVIDUAL@AGENCY*CUSTOMER: message."</code></p>
<p>It was simple and intended to be parsable from our Git logs. However, contribution is about more than just code—and while code contributions are a good place to start, we also needed to ensure that tools for providing attribution and credit would be extensible to non-code contributions as well.</p>
<p>That initial conversation turned into <a href="https://www.drupal.org/node/2288727" rel="nofollow">an issue</a> that spawned lots of child issues and ideas for how best to collect data that allowed a user on Drupal.org to attribute their work to an organization.</p>
<p>The resulting conversations didn’t have a simple solution, but the participation in these issues was lively and full of great ideas. <a href="https://www.drupal.org/node/2323715" rel="nofollow">Determining the format for commit credit</a> is still an active issue.</p>
<p>A commit credit format alone was not going to get the information onto Drupal.org. Eventually, we settled on a couple of different solutions for tracking organization involvement and pulled all of this together into the current state of the Drupal.org Marketplace.</p>
<h2>Organizations that support modules, themes, and distributions</h2>
<p>First, we added a feature to projects on Drupal.org (modules, themes, distributions) that allowed <a href="https://www.drupal.org/node/2267789" rel="nofollow">maintainers to reference organizations that were supporting a module</a>—either by providing time to their developers to give back or by directly funding development.</p>
<p><img src="https://www.drupal.org/files/supporting-organizations-field.jpg" alt="Supporting organizations field" /></p>
<h2>Commit credit user interface</h2>
<p>The next part of the work was <a rel="nofollow">giving maintainers a better way to create their commit message</a> related to a particular issue. A lot of the complexity of deciding who should get credit for work on Drupal.org was simplified into a UI that showed how many people had participated in an issue, how much each person had participated, and whether that person had uploaded files related to the issue. The UI also generated the Git command that would properly tie the resulting code changes to the issue and people involved. The result for a relatively large issue thread looks like the image below:</p>
<p><img src="https://www.drupal.org/files/issues/Screen%20Shot%202014-10-30%20at%201.48.36%20PM.png" alt="Commit credit UI" /></p>
<h2>Attribution</h2>
<p>While that effort made it a lot easier to create commit credits, it didn’t solve the problem of attributing issues that didn’t result in code change, nor did it allow people to give credit to the organizations that made it possible for them to work on that issue—by giving them time or by paying for the work directly as a customer.</p>
<p>To do that we had to figure out a way to add organization attribution to the work being done on an issue. </p>
<p>Issues and comments are the primary tools for tracking what needs to be built out in Drupal code. This is where we determine the features, report the bugs, plan out the work, and more. Tracking intent at the point of this interaction was the best way for us to cover the widest range of contribution to Drupal core, modules, themes and distributions.</p>
<p>The resulting UI for attributing a comment to an organization is simple and straightforward. </p>
<p><img src="https://www.drupal.org/files/initial-comment-attribution.gif" alt="Attribution UI" /></p>
<p>While the instructions for attributing a contribution are documented, there’s still some confusion about when to attribute a comment to an organization versus a customer.</p>
<h3>Attributions for organizations</h3>
<p>These attributions are to allow a person to attribute their participation in a specific issue to a company that allotted time for them to work on the issue. This is typically an employer that pays that person's salary or wage. This organization must be directly tied to the contributor's user profile on Drupal.org as a current or past employer and the organization must have an organization profile on Drupal.org. (<a href="https://www.drupal.org/node/add/organization" rel="nofollow">Create an organization profile</a>).</p>
<p>When you use this feature, you are highlighting an organization that is contributing code—or other improvements through the issue queues—to the community. </p>
<h3>Attributions for customers</h3>
<p>When you attribute a customer, you are essentially saying that the work was requested by that customer but you are not employed by them. Typically this means one of three possibilities:</p>
<ul><li>The customer organization paid the contributor's employer</li>
<li>The customer organization paid the contributor directly</li>
<li>The contributor was volunteering for the customer organization</li>
</ul>
<p>You can select any organization as a customer that has an organization profile on Drupal.org. </p>
<p>When you use the customer attribution, you’re highlighting a different way of contributing to the Drupal ecosystem and often showing relationships between organizations to solve a problem.</p>
<h3>Marking a comment with "I'm a volunteer"</h3>
<p>Making it possible to attribute organizations who have supported contribution also helps us to gather data on purely altruistic contributions. We went round and round on this attribution feature, but in the end, we decided to add “I’m a volunteer” as an explicit attribution option in the attribution UI. It’s deliberately not the default. This is because we want to collect the explicit intent of the users making attributions. While we are now collecting this data, we haven’t yet figured out a great way to show this intent on a contributor's profile.</p>
<p>Possible intent on a comment could include:</p>
<ul><li>I contributed on my own time as a volunteer and community member.</li>
<li>On this issue, some of my time was paid for by an employer, but some of my time was my own.</li>
<li>I contributed this work as a volunteer for one or more customer organizations.</li>
</ul>
<p>The intent of volunteerism is complex, which is why it is hard to add up into neat little data to show on a profile. </p>
<p>Regardless whether they mark "I'm a volunteer" or not, contributors can still be awarded issue credit by a maintainer for their work on an issue.</p>
<h2>Attribution versus issue credit versus commit credit, what's the difference?</h2>
<p>Attribution is the first step of the process to award an issue credit. The contributor attributes their work to one or more organizations and/or as a volunteer. The next step doesn't come until the maintainer of the project (Drupal core, module, theme, distribution) gets involved in the process.</p>
<h3>Issue credits</h3>
<p> <a href="https://www.drupal.org/getting-involved/maintainer/grant-issue-credits" rel="nofollow">Project maintainers may grant issue credits</a> for issues opened against their projects. At any time during the life of the issue, a maintainer can update the credit UI to credit any of the users who have participated in the issue. Any organizations or customers attributed by those users will be credited as well. When the issue is closed fixed, those assigned credits will appear on the appropriate user and organization profiles. </p>
<p>Issue credits are more comprehensive than just code. They can include reviews, designs, prototypes, or just really helpful comments. The maintainer choses what they want to credit even if the issue does not have code associated with it in the form of a patch. This is a great way to help highlight contributions to policy and work to keep Drupal.org running well.</p>
<h3>Commit credits</h3>
<p>Maintainers can <a href="https://www.drupal.org/node/52287" rel="nofollow">grant commit credits</a>. A commit credit gets added to the Git commit message that accompanies a commit by a maintainer of a project on Drupal.org (Drupal Core, modules, themes, distributions, etc.). Sites like <a href="http://drupalcores.com/" rel="nofollow">drupalcores.com</a> track this information in the public changelog and use it to highlight contribution numbers based on the number of times a user is mentioned in a commit message.</p>
<p>Commit credits currently can’t be easily tied to organizations, the parsing would be cumbersome and require a lot of precision when maintainers entered the information into Git, which is a big reason why we needed a UI on Drupal.org to help make all the connections and store that information into a database for easy retrieval.</p>
<h2>Showing contributions on Drupal.org</h2>
<p>Right now, there are three ways that contributions show up on Drupal.org: the user profile, the organization profile, and the marketplace. (We have some ideas to expand this that I will detail below.)</p>
<h3>User profiles</h3>
<p>Do you log in to Drupal.org? Great! You have a profile.</p>
<p>On every user profile, we show the total number of issues that a contributor has helped fix in the past 90 days.</p>
<p><img src="https://www.drupal.org/files/issue-credits-on-user-profile.png" /></p>
<p>You can drill into this information and see the specifics. This is a good way to see a facet of how a person is contributing to Drupal. There are other examples on a user's profile that show other ways they are involved.</p>
<h3>Organization profiles</h3>
<p>Any <a href="https://www.drupal.org/node/1887616" rel="nofollow">confirmed user</a> can <a href="https://www.drupal.org/node/add/organization" rel="nofollow">create an organization page</a> on Drupal.org. Organization profiles are currently related to a single user, but we have plans to roll out additional permissions to relate multiple users to an organization.</p>
<p>On the organization profile, we show the number of issues that an organization has contributed to fixing in the last 90 days.</p>
<p><img src="https://www.drupal.org/files/organization-profile-issue-credits.png" /></p>
<p>Organization profiles also show a list of modules that have been supported by that organization, whether that organization is a <a href="https://assoc.drupal.org/membership" rel="nofollow">Drupal Association member</a> or <a href="https://www.drupal.org/supporters" rel="nofollow">Supporting Partner</a>, and additional self-reported information about their involvement in the Drupal community. </p>
<p>We are currently working to automate more methods for collecting data about how organizations are contributing to the community through <a href="https://www.drupal.org/drupalcon" rel="nofollow">DrupalCon</a> and Drupal Camp sponsorships as well as hosting <a href="https://groups.drupal.org/" rel="nofollow">user groups</a> to help support meetups, and giving to <a href="https://www.drupal.org/global-training-days" rel="nofollow">Global Training Day</a> events. The number of case studies associated with an organization is another indicator of how much they use Drupal and how they are helping get others to use Drupal by sharing their success stories.</p>
<h3>Marketplace</h3>
<p>We recently rolled out some <a href="https://www.drupal.org/news/marketplace-updates-to-highlight-contributing-organizations" rel="nofollow">changes to how the Drupal.org Marketplace is ordered</a> in order to better highlight contributing organizations.</p>
<p>Currently, the marketplace is sorted by number of issue credits awarded to an organization in the last 90 days, then whether the organization is a supporter, then by alphabetical order.</p>
<p>Every organization on the marketplace shows counts of its users on Drupal.org, projects (modules, themes, distributions, etc.) supported, issue credits in the last 90 days, and case studies. We also highlight organizations that are funding the ongoing support and development of Drupal.org by being in our <a href="https://www.drupal.org/supporters" rel="nofollow">supporting partner programs</a>.</p>
<h2>Questions we have heard</h2>
<p>While most people have been excited about the changes to the Marketplace, there have been several questions.</p>
<h3>Why is it only Drupal Service providers?</h3>
<p>The marketplace was originally created to highlight Drupal service providers that are active in the Drupal community. </p>
<p>A "Contributing Organizations" list is the next step to expand the types of organizations we highlight. There are community members that are already working to highlight <a href="https://www.drupal.org/node/2580729" rel="nofollow">customers using Drupal that contribute back</a>, <a href="https://www.drupal.org/node/2581235" rel="nofollow">community volunteers</a>, and volunteers that are working for organizations that rely on volunteerism (many camps fall into this category). </p>
<p>These are great ideas. Contributing to those issues is a great way to help us figure out the best way forward. </p>
<h3>What about "x" contribution type?</h3>
<p>Yes, there are many ways that organizations contribute. We are definitely hoping to add in contributions that we can objectively verify. There are so many great things that organizations can do to support Drupal, both the local and worldwide community, and Drupal.org, including:</p>
<ul><li>Sponsor a DrupalCon</li>
<li>Help plan a DrupalCon</li>
<li>Sponsor, organize, or host a Drupal Camp</li>
<li>Host a Global Training Days event</li>
<li>Host a local user group/meetup</li>
<li>Post a case study of a successful Drupal project</li>
<li>Contribute helpful content to Drupal Planet</li>
</ul>
<p><a href="https://www.drupal.org/node/2649100" rel="nofollow">There are many other phenomenal ways organizations give back</a>. If you know of one that has objective data that can be added to the algorithm we use to highlight organizations, let us know by <a href="https://www.drupal.org/node/add/project-issue/drupalorg?tags=d.o+profile+improvements" rel="nofollow">creating an issue in theDrupal.org Customizations queue</a>.</p>
<h3>Why is it so hard to create an organization profile?</h3>
<p>We definitely want to improve this process. There are over 650 organization profiles on Drupal.org, so we know that people have figured it out, but it could be much more intuitive. </p>
<p>We have also had a several requests to add permissions to organization profiles so that multiple people can be editors of those profiles and help administer who is allowed to be associated with the organization. Our current method of tying users to organization by using an exact text match is a bit clunky. (If you want your organization to list you as an employee, you must exactly enter the title of the organization as it appears on the organization profile and list that organization as "current".)</p>
<h3>What about more tools for individual contributions?</h3>
<p>We have so many <a href="https://www.drupal.org/node/2649100" rel="nofollow">thoughts on how to improve the way we highlight individuals on their profile</a>. We already show issue credits, documentation edits, commits, and lots of self-reported information. There is a lot of objective data to pull from including DrupalCon attendance, whether they have been a speaker at a DrupalCon, translation strings submitted to localize.drupal.org, Drupal Association membership, modules maintained, and many more. </p>
<p>Engagement with the community involves a lot of factors and we want to highlight as many as possible while still making the user profile useful and relatively succinct.</p>
<h2>Help us make Drupal.org better</h2>
<p>In addition to the questions that have come up above, we want your feedback to keep improving the user experience on Drupal.org and make it easier to hightlight the companies that are doing so much to make Drupal the leading platform for the web and the individuals that provide their expertise and commitment to the community. Drop your ideas into the <a href="https://www.drupal.org/project/issues/drupalorg" rel="nofollow">Drupal.org Customizations queue</a>. </p>
2015 Year in Review for Drupal.org2016-01-05T00:00:00+00:00http://joshuami.com/2016/01/05/drupalorg-2015-year-in-review<p><em><a href="https://www.drupal.org/news/2015-year-in-review-drupalorg">Reposted from Drupal.org</a></em></p>
<p>If your 2015 was anything like mine, it passed by extremely quickly. It was marked by periods of frustration (spammers still love Drupal.org) and elation (Drupal 8 launched, woot!).</p>
<p>Below are a few highlights from our 2015 <a href="https://www.drupal.org/news/change-notifications-drupalorg">change logs</a> on Drupal.org. For a more detailed look into each monthly update you can read our <a href="https://assoc.drupal.org/personal-blog-tags/whats-new-drupal.org">“What’s New on Drupal.org?” blog posts</a>.</p>
<h2 id="january">January</h2>
<ul>
<li>New database cluster was put into service and migrations to that server begin</li>
<li><a href="https://www.drupal.org/node/2281763">User and organization profile improvements</a> (lots more work to be done here, but it was a great start)</li>
</ul>
<h2 id="february">February</h2>
<ul>
<li><a href="https://latinamerica2015.drupal.org/">DrupalCon Latin America</a></li>
<li>Launch of Drupal Events (<a href="https://events.drupal.org">events.drupal.org</a>) with the site for DrupalCon Los Angeles - all new events sites are hosted from the same installation</li>
<li>Account creation improvements including a faster 2-step registration that returns users to the site they initiated registration from</li>
<li>Support for the 2015 community elections for board member at large</li>
<li>Implementation of Server Density for systems monitoring</li>
<li>Centralized logging for Druapl.org infrastructure</li>
</ul>
<h2 id="march">March</h2>
<ul>
<li>Relaunch of the Drupal Newsletter and migration of most Drupal.org mailing lists to Mailchimp</li>
<li>Responsive breakpoints added to Drupal.org theme—a way better experience for mobile devices</li>
<li>FTP migration to Fastly</li>
<li><a href="https://www.drupal.org/node/2288727">Issue credits and attribution UI</a></li>
<li><a href="https://www.drupal.org/node/2493801">Advanced Aggregator deployments began to improve performance of the site</a></li>
<li><a href="https://groups.drupal.org/node/463793">DrupalCI sprint at the Drupal Association offices in Portland</a></li>
<li><a href="https://www.drupal.org/node/2446889">“New” indicator for users in their first 90 days to help in welcoming newcomers to the community</a></li>
</ul>
<h2 id="april">April</h2>
<ul>
<li><a href="https://www.drupal.org/node/2481519">Content strategy project complete and recommendations shared with the community</a></li>
<li><a href="https://www.drupal.org/node/2471925">Community user role was deployed and the ability to “confirm” a new user</a></li>
<li><a href="https://www.drupal.org/try-drupal">Try Drupal</a> launches giving evaluators of Drupal an option to quickly try an installation of Drupal provided for free by our hosting partners</li>
<li><a href="https://www.drupal.org/node/2463205">Homepage changes to begin supporting new content strategy</a></li>
<li>Moved all major service deployments from BZR to Git (our standard)</li>
</ul>
<h2 id="may">May</h2>
<ul>
<li><a href="https://www.drupal.org/node/2239973">Two-factor authentication deployed for Drupal.org</a></li>
<li><a href="https://losangeles2015.drupal.org/">DrupalCon Los Angeles</a>
** Presentations on <a href="https://events.drupal.org/losangeles2015/sessions/content-strategy-drupalorg">content strategy</a> and <a href="https://events.drupal.org/losangeles2015/sessions/issue-workspaces-better-way-collaborate-drupalorg-issue-queues">issue workspaces</a></li>
<li>Updates to the log management to show <a href="https://drupal.org/project/usage">statistics from updates.drupal.org</a></li>
</ul>
<h2 id="june">June</h2>
<ul>
<li>Front page changes to highlight community metrics</li>
<li><a href="https://www.drupal.org/node/1815826">“Plan” issue category is deployed</a></li>
<li>Try Drupal refinements</li>
<li>Organization and user profiles updated to show last 90 days of issue credits</li>
<li>DrupalCI goes into production</li>
<li><a href="https://www.drupal.org/supporters/partners">Signature Supporting Partner program launched</a></li>
<li>Updates.drupal.org traffic moved to Fastly CDN</li>
</ul>
<h2 id="july">July</h2>
<ul>
<li>Interns from <a href="http://www.epicodus.com/">Epicodus</a> join the Drupal.org team for 8 weeks</li>
<li>New Git servers configured and deployed to production to replace aging hardware</li>
<li>Core patch testing enabled for DrupalCI</li>
</ul>
<h2 id="august">August</h2>
<ul>
<li>Upgrade of <a href="https://localize.drupal.org/">Localize.drupal.org</a> to Drupal 7</li>
<li>Apache Solr upgrades</li>
<li>Updates to make issues in project queues easier to quickly read and digest</li>
<li>Performance profiling on a new integration server leads to site speed improvements</li>
</ul>
<h2 id="september">September</h2>
<ul>
<li><a href="https://www.drupal.org/node/658048">Drupal.org search improvements</a></li>
<li>First content model update with sections and pages deployed</li>
<li>Solr server high availability improvements</li>
<li><a href="https://barceona2015.drupal.org">DrupalCon Barcelona</a></li>
<li>Marketplace updates to order organizations by last 90 days of issue credits and supporter status</li>
</ul>
<h2 id="october">October</h2>
<ul>
<li>Drupal 8 RC1</li>
<li>DrupalCI becomes the primary test platform for Drupal 8 Core; legacy testing is disabled</li>
<li>Flag module and administrative views are deployed for a new take on spam fighting</li>
<li>Drupal 6 & 7 testing are enabled on DrupalCI</li>
<li>Kicked off the first <a href="https://www.drupal.org/node/2589167">Drupal.org membership drive</a></li>
<li>Breadcrumbs for top level sections to make navigation a bit easier</li>
</ul>
<h2 id="november">November</h2>
<ul>
<li><a href="https://www.drupal.org/node/1933988">Server-side support for contributed Drupal 8 translations on Localize.drupal.org</a></li>
<li>Drupal.org development environments now build in under 20 minutes</li>
<li>All sites are <a href="https://www.drupal.org/news/drupalorg-migrates-content-and-file-delivery-to-fastly">migrated to Fastly for CDN services</a></li>
<li>Behind-the-scenes config and code changes for the new Drupal 8 area of the site (<a href="https://www.drupal.org/8">drupal.org/8</a>)</li>
<li><a href="https://www.drupal.org/news/drupal-8.0.0-released">Drupal 8 successfully launched</a></li>
<li>Drupal 8 section built with a new theme</li>
<li>Theme changes to the front page and header associated with Drupal 8 launch</li>
<li>Download experience changes</li>
<li>Began testing Distil as a solution to reduce spam account creation on Drupal.org</li>
<li>3 new test environments added to DrupalCI</li>
</ul>
<h2 id="december">December</h2>
<ul>
<li><a href="https://www.drupal.org/node/2551607">Published plan for Composer support on Drupal.org</a></li>
<li>Old testbots are disabled and all Drupal testing is now run through DrupalCI</li>
<li>Homepage changes to support the membership campaign</li>
<li>Presented the new community engagement plan to the board and working groups</li>
</ul>
<p>That was a lot of work in what was a tumultuous year. Between all the DrupalCon websites, and major updates to some of our existing sub-sites, we launched as many sites as some web development shops—at least the smaller ones.</p>
<p>We made several small but high impact usability improvements to Drupal.org, and built better features to highlight the contributions of individuals and organizations on their profiles. We modernized and hardened our infrastructure. We made Drupal.org a little more beautiful to better promote this amazing software and community that we support to people finding us for the first time. Among all of that…</p>
<h2 id="what-was-the-most-important-work-in-2015">What was the most important work in 2015?</h2>
<p>If I had to call out just one thing the Drupal.org team did this year that was essential to the success of Drupal and the community, I would highlight the great work done to launch DrupalCI and make it a production system that could handle our community’s unique testing needs.</p>
<p>DrupalCI (the CI stands for Continuous Integration) started as a community initiative. It had been in progress for years as work to replace our aging testbot infrastructure. The old testbots required a manual spin up of new testing servers that were hosted on VMs in a cluster at the Open Source Labs. They were proprietary and rigid. If a testbot went down, it would require manual intervention to free up the queue and allow tests to begin again.</p>
<p>By working closely with the community throughout Q2 and Q3 of 2015, we were able to launch a testing infrastructure that supports multiple testing environments—a feature which has already helped support other projects, like PHP 7—dynamically scales to the testing load, and which is tightly integrated with the Drupal.org issue queues. Overall the new system is much more configurable and far more scalable than the previous system.</p>
<p>Why is testing so vital to the community? Because Drupal 8 represents a much larger code base than Drupal 7. A huge proportion of those new lines of code are tests. Every time a patch is submitted to Drupal Core, up to 15,000+ tests are run—with over 100,000 assertions. Core maintainers and initiative leads need those tests to help them understand how the new code introduced will affect Drupal. Contrib developers can also use DrupalCI to ensure their modules will work well with Drupal.</p>
<p>Though the work to get DrupalCI into production and optimized, we were able to give core maintainers faster and more reliable information about code submitted for inclusion in Drupal.</p>
<p>I honestly believe that without DrupalCI, we would not have had Drupal 8 in 2015. It sped the release of Drupal, which makes it the most accomplishment for our team in 2015.</p>
<h2 id="happy-but-not-satisfied">Happy but not satisfied</h2>
<p><a href="http://www.garfieldtech.com/blog/drupal8-happy-not-satisfied">Larry Garfield (Crell) wrote a blog post</a> shortly after the release of Drupal 8 where he talked about how he was happy that Drupal 8 was released—and feels it is a huge leap forward for the project—but he is not satisfied because he sees how much work is left to be done.</p>
<p>That is the nature of software. It can always be a little better—more performant, more usable, more extendable.</p>
<p>Drupal.org improved in 2015, but the team is far from satisfied. There is so much more work to complete.</p>
<p>At DrupalCon Amsterdam, I outlined the <a href="https://docs.google.com/a/association.drupal.org/presentation/d/1wRDKL6gwLhPpmnY4XV0XnrGBVavVgJaUxwfk-LDLHEc">top 7 initiatives that the Drupal.org team would focus on</a>. When I look back on the year, we did not hit all of our goals.</p>
<p>In April, one year into my time at the Drupal Association, I could point back to a lot of accomplishments, but for many, the improvements were not fast enough. By DrupalCon Los Angeles, we had achieved one of our initiative goals, and moved forward many more, but we added four more to our list. In agile project management terms, we burned up (added more work) than we burned down (work completed).</p>
<p>While the fact that more work is requested faster than we could complete it is not unusual, it did make us think, and we learned some lessons. We have to focus on fewer, high impact priorities. We need to plan for the unplannable - we know there will be unforeseen needs from the community and new technology to support that may not exist yet, and we must be flexible enough to respond. We need to build more partnerships and find great solutions with technology providers that are experts in their fields.</p>
<p>We are pretty excited that much of the planning and design work put into 2015 should result in a much more rapid pace of change to Drupal.org in 2016.</p>
<h2 id="whats-next-for-drupalorg">What’s next for Drupal.org</h2>
<p>So, we are celebrating all the good things from 2015—there were a lot—but we are also closing the book on what was a challenging year.</p>
<p>Our focus shifts now to supporting the community as we all work to make Drupal 8 successful. We’ll be keeping the <a href="https://drupal.org/roadmap">Drupal.org Roadmap</a> up to date and adding in new initiatives.</p>
<p>As we start the new year, we are still committed to the <a href="https://www.drupal.org/node/2481519">content strategy</a> work that will make the content creation experience on Drupal.org and sub-sites better. This will improve our documentation as well as make it easier to talk about the benefits of Drupal to decision makers that help choose Drupal as the best content management framework for their organization.</p>
<p>Additionally, we are doing some <a href="https://www.drupal.org/node/2551607">exciting work to better support Composer on Drupal.org</a> to align us with the rest of the PHP community as well as planning some much needed improvements to our Git workflow and developer tools.</p>
<p>Lastly, we are not done improving how we highlight contributions within the Drupal community. <a href="https://www.youtube.com/watch?v=4NN5EM4CYVE">As Dries outlined in Amsterdam</a>, Drupal is a public good. We need to highlight the great work that people around the world are doing in building Drupal and the community that supports it.</p>
<p>We can’t wait to get started on 2016!</p>
Drupal.org Migrates Content and File Delivery to Fastly2015-11-16T00:00:00+00:00http://joshuami.com/2015/11/16/drupalorg-migrates-content-and-file-delivery-to-fastly<p><em><a href="https://www.drupal.org/news/drupalorg-migrates-content-and-file-delivery-to-fastly">Reposted from Drupal.org</a></em></p>
<p><img src="https://www.drupal.org/files/fastly-logo_0.png" alt="Fastly logo" class="right" width="250" />We are so stoked to announce our partnership with <a href="http://fastly.us/1H1mcCF">Fastly</a>. Fastly is now serving up all of our traffic from the *.drupal.org domain Drupal sites and related services.</p>
<h2>Drupal.org is big</h2>
<p>Drupal.org and its services handle over <em>1.5 billion</em> requests per month; this is a massive amount of traffic for an open source project.</p>
<p>Every time cron runs the update service on your Drupal site, it talks to updates.drupal.org. Every time you download a copy of Drupal or any projects on Drupal.org, you talk to ftp.drupal.org. (We see over 400,000 downloads of Drupal core in a typical month—way more around DrupalCons and major community sprints.)</p>
<p>Each month we have over 15 million unique pageviews on Drupal.org—by over 2 million unique visitors.</p>
<p>All of those stats are about to rise significantly with the launch of Drupal 8. Drupal 7 caused a 30% bump in traffic when it was released and we expect even more with the launch of Drupal 8.</p>
<h2>Fastly is… well… fast</h2>
<p>In 2014, we implemented a CDN (content delivery network) for Drupal.org. The impact was immediate. Everything was faster. We met our initial goal of getting Drupal.org sites and services fronted by a CDN.</p>
<p>That CDN solution was a good start, and while it lacked features we didn’t know we needed, it improved our ability to deliver Drupal-generated content as well as packaged projects.</p>
<p>Late in 2014, we were introduced to Fastly. They offered us an opportunity to try out the service for ftp.drupal.org and we haven’t looked back.</p>
<h3>Varnish for the win!</h3>
<p>You may already use the open source Varnish for your Drupal sites. Varnish is an HTTP accelerator. Drupal.org began using Varnish in 2009 to reduce load on its web servers. It is a powerful cache that likely sits in front of your web origins. Fastly gives us a globally distributed Varnish cache with all of the features we are familiar with.</p>
<p>Fastly hires maintainers of the Varnish project and are helping move it forward. Drupal 8 cache keys are also a straight correllation to Fastly Surrogate Key purging and Fastly’s active involvement is another great <a href="https://www.drupal.org/node/2491561">example</a> of Fastly working with an open source community (ours!) to build a better experience.</p>
<p>Yay, open source!</p>
<h3>Open Source Alliance</h3>
<p>Speaking of open source, <a href="http://fastly.us/1LewNVK">Fastly has an open source alliance</a> that provides free content delivery to projects like Debian, the MIT Media Lab, the W3C, Memcached, Linux Foundation, and more. They have even open sourced some of their technology stack to make it easier for others to build systems with their tools. We love partnering with these sorts of organizations.</p>
<h3>Supporting Technology Partner</h3>
<p>Speaking of partnering, Fastly took their partnership a step further and are helping fund a lot of great work on Drupal.org through commitments to sponsor DrupalCons and even the release of Drupal 8. Look for them at upcoming DrupalCons. Thanks Fastly!</p>
<h2>How Fastly Works</h2>
<p>Fastly’s <a href="http://fastly.us/1kv5OAw">Global Network</a> consists of Points of Presence (POPs) running Fastly’s custom Varnish software stack. These POPs are spread around the world in strategic locations close to the highest density Internet Exchange Points, ensuring cached content is just a few milliseconds away.</p>
<p><img src="https://www.drupal.org/files/fastly_global_delivery_network_pop_map_v25_11.05.15.png" alt="Global network" /></p>
<p>Fastly has a powerful feature that allows us to specify a specific POP as a shield to our origin server. In our case we chose the Seattle POP because of its proximity to our servers at the <a href=”http://osuosl.org”>OSL</a> in Corvallis, Oregon in the United States.</p>
<p><img src="https://www.drupal.org/files/caching-basics-1st-request-withshielding.png" alt="Caching basics" /></p>
<p>The origin shield configuration means all requests to Drupal.org flow through Fastly’s origin server in Seattle before reaching our Drupal.org origin. If <a href="http://fastly.us/1NvHzsE">Fastly’s origin server</a> in Seattle has already cached the file, no request to our Drupal.org origin is necessary.</p>
<p><img src="https://www.drupal.org/files/caching-basics-subsequent-request-withshielding.png" alt="Caching basics, subsequent requests" /></p>
<p>In practice, this allows us to deliver a huge amount of content with an extremely high hit ratio. (See that spike in the GIF below, that’s one of those regular times that cron jobs request a ton of content from our updates server. Fastly is just churning along.)</p>
<p><img src="https://www.drupal.org/files/fastly-updates-performance.gif" alt="Really cool animated GIF showing updates traffic" /></p>
<p>(Fastly made us confident we could put this enormous GIF in our write up.)</p>
<h3>Faster changes, faster response time</h3>
<p>Another advantage to Fastly has been the efficiencies it has given us by allowing us to quickly make changes to how they cache our content. With our previous CDN, changes could take up to 4 hours to propagate through the system. We can now change a Varnish config and reflect those changes in under 5 seconds. That has been a huge help when we need to make a hot fix to production and keep Drupal.org delivering Drupal to the world.</p>
<h3>Logging and improving our usage stats</h3>
<p>Fastly also allows us to <a href="http://fastly.us/1lnEQed">stream our logs</a>. This means we can parse those logs and turn them into data. We are already seeing much more reliable project usage statistics with from our new log processing for updates.drupal.org thanks to Fastly’s log streaming.</p>
<p>More accurate logs mean we can provide users with better data to help them to make better decisions about modules, themes and distributions that have the most installations.</p>
<h2>Next Steps with Fastly</h2>
<p>We have been impressed with Fastly, both for their responsiveness and the performance improvements we have seen while using their services. They have given a ton to our community and we can’t wait to see where they take Varnish and their open-source-powered service. If you are interested in using Fastly with your own Drupal sites, you can <a href="https://www.drupal.org/marketplace/fastly">learn more about Fastly</a> on their Drupal.org organization page or sign up for a <a href="http://fastly.us/1iZPKoy">developer account at Fastly.com</a>—there is even a <a href="https://www.drupal.org/project/fastly">Fastly module</a> for Drupal7, and one for Drupal8 is on the way!</p>
Digital literacy - the collected media of our lives2015-10-24T00:00:00+00:00http://joshuami.com/2015/10/24/digital-literacy<p><a href="https://medium.com/@joshuami/digital-literacy-the-collected-media-of-our-lives-b818b3c5703c">Republished on Medium</a></p>
<p>My My kids are getting older. They are fully teenagers at this point. My son has a beard! My daughter gets mistaken for her mother! I’m not feeling that old—we started parenthood quite young compared to our peers—but I do have to think about things like life insurance and paying for college.</p>
<p>It strikes me that I am part of the first generation of humans that really has to think of the legacy they leave their kinds in terms of the digital footprint we leave behind. My son is just old enough that his early years were documented on film—which means they are sitting in a box somewhere. My daughter’s entire life has been digital—which means her early years are sitting on a hard drive.</p>
<p>Just like my son’s first three years, photos were once kept in boxes and albums and stored in a trunk or on a shelf or in an attic.</p>
<p>There was a time when the music of your life was a collection of records or tapes or CDs that sat in shelving next to your home stereo.</p>
<p>Your home videos and favorite movies were VHS tapes or DVDs or BlueRay disks that lined a shelf next to your TV—personally I skipped BlueRay and went straight to HD digital files.</p>
<p>Books were… well… books. Your library of books may now be a mixture of hard copies and an increasing number of ebook titles that you access from your ereader, tablet or smartphone.</p>
<p>Now all of that personal history and media is stored in a digital format. For many, especially the generation born after the turn of the century, there is likely little physical evidence of their lives outside of the digital records kept by their parents or by Facebook, Instagram, etc.</p>
<p>This digitization of life is not all bad. Where a fire or flood would have once wiped out your personal collections, you can with relatively little effort redundantly store this data in a cloud provider of some sort.</p>
<p>If you are under 25, you probably think of your music collection as something that lives on your phone—if you even collect music.</p>
<h2 id="do-you-really-need-to-own-all-of-that-stuff">Do you really need to own all of that stuff?</h2>
<p>For many, ownership is optional. “Ownership” is a first world problem. Once we own things, we have to figure out what to do with all our stuff. We pack it onto shelves and into boxes. For urbanites and the growing population of digital natives. All that space used for media seems a bit of a waste.</p>
<p>Many of us are turning to digital services rather than physical copies. Though arguably, I know quite a few people who are turning to the retro and the analog in an attempt to connect with their media in a different way. They can do this because they have the expendable income and interest. It is really more a hobby than a true return to the past. (None of them are giving up their smartphones.)</p>
<p>With online services, you are giving up that tangible ownership in exchange for flexibility of access. Sometimes you are even giving up a specific and limited library in exchange for access to a broader and more comprehensive curated collection. Few people have the financial resources to collect the contents of Netflix or Spotify or Pandora.</p>
<p>I have a friend with a massive physical collection of music and movies. It is more content than could be reasonably watched or listened to in a lifetime. He tries, but really it amounts to a huge expenditure on a collection that will likely only be listened to or watched once. He could have rented every title in that collection for about a third of the purchase cost.</p>
<p>If you have ever moved, you can appreciate that a smaller footprint makes that process easier. You can buy or rent less space if your collections are primarily digital. You might even be able to argue you have a smaller environmental footprint. (Though if you are running electronic devices 24-7, you may be chipping away at that environmental benefit pretty significantly.)</p>
<p>So you decide you don’t need all of those physical trappings. What are you giving up?</p>
<h2 id="multiple-ways-that-you-have-lost-ownership-of-your-media">Multiple ways that you have lost ownership of your media</h2>
<p>In the example of my friend with the huge media collection, he owns every one of those titles. If he chooses, he can give them away, sell them, hand them down to his child as an inheritance. He owns the physical representation of all of that artistic endeavor. (His son—a digital native of 16—doesn’t want all that media, so the value of passing it to an heir is a bit suspect.)</p>
<p>Most people did not realize they gave up ownership of their music and movie libraries when they moved to a digital service hosted in the cloud.</p>
<p>Whether you are using Amazon or Apple, Netflix or Spotify, you don’t really own your digital content anymore. Essentially you are leasing your content for a very long time. Your lease of the content is directly tied to a digital identity.</p>
<p>This digital identity is a key part of digital ownership. What happens if you lose your password and access to the email account associated with the collection? You lose your lease. This would be particularly common at the point that someone dies. Their friends and loved ones could pick up the subscription costs or digital rights of downloaded media only if they have shared the password to the account or the email account that was tied to the account for password retrieval.</p>
<p>If you enabled two-factor authentication—and you should for security’s sake—transferring those digital rights is even more complex. Did you put your recovery codes in a place your loved ones know about?</p>
<p>This issue of ownership extends to your the photos and videos—your personal creations—if you have no backups of these files on media you can access outside of the service. Have all of your photos in an online service? What happens if that service loses your data? What happens if you die and your children lose access to the entire digital documentation of their lives? Have all your photos on your phone, but you never backup your phone? What happens if it is stolen, damaged, or lost? This is the digital equivalent of your physical possessions being lost to fire or disappearing on a sinking ship while being transported across the ocean.</p>
<h2 id="a-sane-way-to-protect-your-digital-collection">A sane way to protect your digital collection</h2>
<p>The thought of losing all your stuff may scare you a little. You may wonder what you should do to protect that precious personal history. You don’t want to disappear and be forgotten because your children have no history to pass to their children—right?</p>
<h3 id="your-grandkids-dont-want-that-photo-of-a-flower-you-took-on-a-walk-through-the-neighborhood-when-you-were-27-unless-it-was-really-freakin-spectacular">Your grandkids don’t want that photo of a flower you took on a walk through the neighborhood when you were 27… unless it was really freakin’ spectacular</h3>
<p>First, get over yourself. Four generations ago the idea of passing significant amounts of photo or video documentation was not possible. You might have a photo or two of your great-great grandparents, maybe an antique or two that was passed down from family member to family member. That scarcity has the effect of making those items more valuable, more precious.</p>
<h3 id="save-the-best-and-give-it-a-little-extra-attention">Save the best and give it a little extra attention</h3>
<p>So think about what you feel has the most value to pass down. I have over 200GB of family photos and home videos. I do not want my family to have to wade through those 200GB of data to decide what is worth keeping, so I’m not necessarily going to give it the same level of care in backups.</p>
<h3 id="rate-your-data">Rate your data</h3>
<p>In IT, information owners tend to rate data by its importance. Do the same with your digital collections. You should be backing up the most precious and valuable memories and data in at least two ways.</p>
<h3 id="rotate-your-backups">Rotate your backups</h3>
<p>Make sure that you regularly rotate that data to someplace safe and away from your computer. That could be a copy on a portable hard drive that you keep in a safe deposit box, but chances are you are not going to update that very often. A fire safe in a detached garage might work, but again, you are probably taking a risk.</p>
<h3 id="be-redundant-with-your-cloud-storage">Be redundant with your cloud storage</h3>
<p>If you can whittle that collection of precious down, you will be best off if you select a couple of online services to backup your data. Put a copy in Dropbox or Amazon Web Services. Backup that photo library to both Google Photo and Flickr. Do not trust a single point of failure.</p>
<h2 id="think-about-what-it-means-to-live-in-a-digital-age">Think about what it means to live in a digital age</h2>
<p>It is far too easy to ignore just how much our lives are reliant on vast amounts of storage sitting in data centers around the world. We are a little out of touch with the past and a little oblivious to what the future may hold. We are the first generations in human history to be burdened by the cognitive load of digital living.</p>
<p>That doesn’t make us all that special. Every generation has lived through the significant history that defines it, but we do carry a little extra responsibility to our kids and loved ones to pass on data in a somewhat intentional way.</p>
<p>And I really hope that my great-great grandkids don’t judge me by that photo that I took while on a walk while I was 27. I was just trying to learn how to use that shiny new digital camera.</p>
Marketplace Updates to Highlight Contributing Organizations2015-09-22T00:00:00+00:00http://joshuami.com/2015/09/22/marketplace-updates-to-highlight-contributing-organizations<p><em><a href="https://www.drupal.org/news/marketplace-updates-to-highlight-contributing-organizations">Reposted from Drupal.org</a></em></p>
<p>We are excited to announce some big changes to the <a href="https://www.drupal.org/drupal-services">Drupal.org Marketplace</a>. In <a href="http://amsterdam2014.drupal.org/session/keynote-dries-buytaert.html">Dries’ Amsterdam Keynote</a>, he made a compelling case for showing the contributions of organizations that are helping build Drupal. By highlighting organizations that give their employees time to give back, we make it possible for more people to give time to making the project better.</p>
<p>In March, we took steps to begin collecting this information by allowing individuals that were contributing in the issue queues to attribute their contributions to organization that they are employed by or customers that funded the work. When a maintainer of a project (module, theme, distribution or Drupal Core) closes an issue as fixed, they have an opportunity to <a href="https://www.drupal.org/node/2495315/">pass on credit</a> to the individuals who helped contribute to fixing the issue—and not just code contributions, but any kind of feedback, review, designs, etc.</p>
<p>We called this system issue credits and it has been a huge success. We now show the last 90 days of issue credits awarded to an individual or organization on their profile.</p>
<p>Today, after months of collecting this data, we are taking how we highlight contributing organizations to a new level.</p>
<p>With this launch, we are removing the distinction of “featured service providers” versus “all service providers”. By using data about these organizations contributions, we can provide a single list of all organizations ordered by their contributions.</p>
<p>For now, we are using issue credits as the primary sort. The secondary sort highlights organizations that are giving back by supporting Drupal.org through the supporting partner program or organization membership. Soon, we plan to incorporate case studies submitted, DrupalCon sponsorships, and camp sponsorships to help make a more complete picture of how organizations are contributing to our community.</p>
<p><a href="https://www.drupal.org/drupal-services">Give it a look</a> and give us your feedback.</p>
Keeping sharp with Jekyll for blogging2015-07-08T00:00:00+00:00http://joshuami.com/2015/07/08/keeping-sharp-with-jekyll-blogging-tool<p>To keep my skills sharp, I have started a series a Friday experiments. This Friday, I decided to explore moving my personal site—which was moved to Drupal 7 last year off of a Wordpress installation—to a Jekyll installation with Github Pages integration.</p>
<p>Another key reason to test the Jekyll experience is that it has become a frequently used tool in creating headless Drupal installations. (Headless is something I am very keen to try at some point.)</p>
<p>To pull this together I used the following resources:</p>
<ul>
<li><a href="http://joshualande.com/jekyll-github-pages-poole/">Joshua Lande’s excellent run down</a> of how he migrated his personal blog</li>
<li><a href="http://jekyllrb.com/docs/home/">Jekyll’s complete—but rather esoteric—documentation</a></li>
<li><a href="https://help.github.com/articles/using-jekyll-with-pages/">GitHub’s documentation on using Jekyll with GitHub Pages</a></li>
</ul>
<p>Instead of the default <a href="http://getpoole.com/">Poole</a> layouts that Lande used, I decided to go with <a href="https://github.com/poole/hyde">Hyde</a>.</p>
<p>So far, I’m enjoying typing a post directly into markdown rather than a bloated WYSIWYG. We’ll see if that encourages me to write a bit more.</p>
Drupal Association CTO - Year 12015-04-08T00:00:00+00:00http://joshuami.com/2015/04/08/drupal-association-cto-year-1<p><em><a href="https://assoc.drupal.org/blog/joshuami/drupal-association-cto-year-1">Reposted from Drupal Association Website</a></em></p>
<p>My first day as Chief Technology Officer for the Drupal Association was the 31st of March 2014. I like to joke that I started a day before April Fool's Day on purpose.</p>
<p>As the first CTO for the Drupal Association, I'd like to highlight a few of the lessons I've learned and the accomplishments of the Drupal.org Product and Engineering Team over the past year.</p>
<h2>
Learning</h2>
<p>In my first week... I learned... and learned... and learned. Listening is an important skill for any leader, but it is never more important than when you are picking up the 13-year history of the website you are inheriting as your responsibility.</p>
<p>I have been actively building and managing teams that build with Drupal since the early days of Drupal 6. Even before beginning to use Drupal, I've been focused on growing product and engineering teams that build big complex sites. It excited me to get involved with Drupal, doing what I feel I do pretty well, at a whole new level. I felt I knew a lot about the software, but had a relatively shallow experience with the site that powers the community.</p>
<p>In that first week, I learned that I had an amazing team of four—two of which were new to the team themselves—to help me learn the ropes. I also learned that they were overworked, more than overbooked, and still recovering from the Drupal.org upgrade to Drupal 7 that occurred in the fall of 2013.</p>
<p><img alt="Drupal.org ecosystem of sites and services" src="https://docs.google.com/drawings/d/1Y7KkP0IgISsN8LuaAnJPKlxUxBe3883_oFS36It82vg/pub?w=954&h=563" style="height: 366px; width: 620px;" /></p>
<p>I learned that Drupal.org is not built solely on Drupal. It is an intricate combination of technologies of which Drupal is only a fraction of that whole. Drupal.org is over 16 websites, CDN services, Git repositories, some Python in interesting places, some Puppet and some Jenkins and so much more. The volunteers that built our infrastructure were—and continue to be—amazing. They also have very, very understanding employers.</p>
<p>I learned that the Drupal Association is a phenomenal group of professionals (16 when I started) that are truly committed to our open source community.</p>
<h2>
Expect the worst, but assume the best</h2>
<p>In my second week, we bled—not literally, mind you. Heartbleed was my trial by fire. It was an excellent opportunity to meet key infrastructure and security team volunteers. They are an amazing group of professionals.</p>
<p>In that process of securing Drupal.org from a threat that was rocking the Internet, I found the mantra that would come to define how to look at a site the size and scale of Drupal.org. Expect the worst, but assume the best.</p>
<p>Whether it be security, spam, or regressions on deployment, the best way to make sure your site can respond is to expect the worst possible behavior by untrusted users. At the same time, we are an open source community that aspires to be inviting and to grow. We have to assume every new user of Drupal.org is a potential future Drupal contributor that wants to make us better. That is quite the dichotomy to operate within as a team of technologists.</p>
<h2>
Grow, grow, grow</h2>
<p><img alt="Drupal.org team growth in 2014" src="https://docs.google.com/drawings/d/1_XXRDZhRjOk8R1QKHkcwRrBm6l37m0RKxzORj2ACCYQ/pub?w=960&h=540" style="width: 620px; height: 349px;" /></p>
<p>I mentioned that when I started, I had a team of 4 in an organization of 16. The Drupal.org product and engineering team is now 11.5 members strong—and the association just hired its 34th employee. That is incredibly rapid growth for a small organization to go through in one year.</p>
<p>Over the past year, we have spent a lot of time setting or norms and forming our culture. As a developing team, we had to pick our tools for communication and project management. We implemented new processes for defining the priority of our work. We established patterns of communication to make sure we regularly involved in the community—working groups in particular—in our iterations of getting things done. We have also been an incredibly productive team for one forming so quickly.</p>
<h2>
Governance, communication and the community</h2>
<p><a href="https://www.drupal.org/governance/drupalorg-working-groups/">Governance for Drupal.org</a> was established in early 2013. When I started, a good portion of my first few months was figuring out how to integrate myself and the team I was building into the working groups.</p>
<p>These working groups had gone through a couple of years worth of ideation processes and had a strong communication focus with their work. What the working groups did not have was enough sustained volunteer contribution and support to build the tools they were identifying as a need. Additionally, they were in a tough place where they had the authority to make decisions, but they didn't necessarily have a way to make sure those decisions were implemented.</p>
<p>My goal since beginning this work was figuring out a way to get all the cool community ideas implemented faster. A the same time, it is also my role to figure out how to make Drupal.org and the tools and infrastructure our community funds through the Drupal Association sustainable. These tools have to help us grow our community and help increase the skills of the learners in our community that will one day become the experts in our community.</p>
<p>My team's biggest challenge is continuing to make our ecosystem better as fast as we can while letting the community know where we are focusing our energies and getting the community to get involved when they have the time and interest.</p>
<p>With the guidance of the working groups and the Drupal Association Board, we were able to establish a <a href="https://www.drupal.org/roadmap">strategic roadmap</a> that helps communicate what is coming next for Drupal.org.</p>
<h2>
Getting things done</h2>
<p>So what have we done in the past year? While growing a team and building new norms and processes, we were able to accomplish quite a lot...</p>
<h3>
Support for semantic versioning</h3>
<p>We updated Drupal.org processes to support semantic versioning. 8.0.0 here we come! This also means we will be able to have more frequent Drupal releases reducing the time between new versions—8.1.0, 8.2.0 on deck.</p>
<h3>
Better infrastructure and deployments</h3>
<p>We improved page load times—doing this included changes such as CDN-fronting our infrastructure and upgrading hardware that was long overdue for a refresh. The infrastructure is amazing, but there are few volunteers that understand it enough to keep maintained at the level the community needs. We now have dedicated DevOps engineers that are making sure our technology stack is performant and stable.</p>
<p>There are over 400 behavior driven design (BDD) tests that now allow us to more confidently make deployments to Drupal.org.</p>
<p>And we make a lot of deployments to Drupal.org... about 68 a month to just Drupal.org customizations. (About 3 commits per month are volunteers with the balance made by staff.)</p>
<p>When we deploy big stuff (there is a lot of small stuff), we let people know about it. The <a href="https://www.drupal.org/change-notifications">change notification process has a subscription option</a> and we post those notices to our twitter handle at @drupal_infra.</p>
<h3>
Credit for organizations that contribute</h3>
<p><img alt="Animation of the organization attribution UI." src="https://www.drupal.org/files/issues/comments_0.gif" style="width: 620px; height: 382px;" /></p>
<p>Recently, users were given the ability to <a href="https://www.drupal.org/node/2340363">attribute their comments in the issue queue to the organization</a> that gave them the time to make the contribution (their employer) or paid for the contribution (a customer). That is a huge leap for our community. It is going to give us a path to tracking how Drupal core, contributed modules and themes are made possible by the awesome organizations that are using Drupal.</p>
<p>There is an excellent UI for maintainers to give credit to the users can commit, and we are expanding this to allow that credit to extend to the organizations involved.</p>
<h3>
Improved account creation and new user experience</h3>
<p>We have done a huge amount of work making creating a new account easier for users—and harder for spammers. We made it much easier for a user to create their Drupal.org profile from a subsite. These were important steps toward better user profiles.</p>
<p>A label now appears beneath the user picture of new users for the first 90 days they are on Drupal.org to help us welcome them and get them involved.</p>
<p>Coming soon, users that have contributed significantly to Druapl.org will get the "community" role on Drupal.org will be able to confirm new users to make it easier for them to post. This will make it easier for sprint organizers to help us engage with new users</p>
<h3>
User profiles are getting better</h3>
<p>User profile pictures have been implemented—which seems small, but it has a big impact in a comment thread. You now have a decent URL to hand out for your Drupal.org profile (e.g. <a href="https://www.drupal.org/u/joshuami">drupal.org/u/joshuami</a>). Mentor fields also show the pictures of mentors. We made it easier for us to synchronize data between Drupal.org subsites.</p>
<p>By moving our crediting system to issue comments—rather than just commits—we are expanding what the definition of "contribution" is for our community.</p>
<p>With better data about our contributors, we'll be able to better highlight how people are involved.</p>
<h3>
Launch of Drupal Jobs</h3>
<p>In August of 2014, we launched <a href="https://jobs.drupal.org">Drupal Jobs</a> to help connect Drupal employers with job seekers. It continues to grow and we are now up to over 1,000 job seekers. We average over 200 open positions per month on Drupal Jobs.</p>
<h3>
Responsive updates to Bluecheese (Drupal.org's theme)</h3>
<p>As a special New Year's present to the awesome volunteers that helped add responsive elements to our theme, our team merged those changes to the Bluecheese theme making Drupal.org much easier to use on mobile devices.</p>
<h3>
Launching and maintaining 3 Drupal Cons sites and the new Drupal Events</h3>
<p><a href="https://austin2014.drupal.org">DrupalCons Austin</a>, <a href="https://amsterdam2015.drupal.org">Amsterdam</a> and <a href="https://latinamerica2015.drupal.org">Latin America</a> all had sites built on Drupal to launch, maintain and archive. <a href="https://losangeles2016.drupal.org">DrupalCon Los Angeles</a> is the first site on our new <a href="https://events.drupal.org">Drupal Events</a> platform that is totally changing the way we approach our ticket and registration process. Also, this site will allow us to maintain a single living archive of all Con session presentations and profile data.</p>
<h3>
User research</h3>
<p><img alt="Dreyfus Model of Acquisition modified to Drupal.org user personas" src="https://docs.google.com/drawings/d/1UaeHwz3mfbLRQrQNCRnFG_s-kTX_gYGyBehYu7VT5c4/pub?w=960&h=540" style="width: 620px; height: 349px;" /></p>
<p>Starting at DrupalCon Austin, working with user research coach Whitney Hess, the team interviewed more than 30 community members of all types. That research was turned into the research that made up our skills acquisition model for our new personas.</p>
<h3>
Content strategy</h3>
<p>Forum One was selected to help us build out a new content model, site map and governance plan for Drupal.org.</p>
<h2>
Next steps</h2>
<p>The summary above does not come close to covering all of the great work. The Drupal.org product and engineering team has an awesome roadmap planned for the coming months.</p>
<ul>
<li>
More profile improvements for users and organizations</li>
<li>
New content model and governance from the content strategy work</li>
<li>
Drupal.org style guide and design system</li>
<li>
DrupalCI - next version of continuous integration testing for Drupal.org</li>
<li>
Issue workflow improvements to make contributing and maintaining easier</li>
<li>
Better search and discovery</li>
<li>
Improved data to help find and select modules, themes and distrobutions</li>
<li>
Updates to Drupal Groups (groups.drupal.org) and Drupal Translations (localize.drupal.org)</li>
</ul>
<h2>
Thank you</h2>
<p>To the supporting partners that made all of this work possible, thank you. To the community volunteers that gave of their time to contribute code and ideas and feedback, thank you.</p>
<p>One last thank you, I feel absolutely blessed to work with such an awesome team of designers and developers, a project manager, a product manager and some incredible working group members.</p>
<p>I cannot wait to see what the next year brings.</p>
Content Strategy Update: What are we up to?2015-02-12T00:00:00+00:00http://joshuami.com/2015/02/12/content-strategy-update<p><a href="https://www.drupal.org/news/content-strategy-update-february-2015">Reposted from Drupal.org</a></p>
<p>Back in December, as part of our ongoing efforts to improve Drupal.org, we <a href="https://assoc.drupal.org/blog/tvn/forum-one-work-content-strategy-drupal.org">kicked off a content strategy project with Forum One</a>. Drupal Association engineering and marketing/communication staff partnered with the <a href="/governance/drupalorg-working-groups/content">Drupal.org Content Working Group</a> and met for a two-day workshop to help get the project team from Forum One (content strategists and user experience designers) up to speed on Drupal.org and the ecosystem of sites and services that our community uses to build and use Drupal.</p>
<p>Over the past month, we have pulled together many detailed documents to help guide our work. While we are only about halfway through this project, we wanted to share a bit of the work-in-progress that will influence Drupal.org’s content strategy in the coming months.</p>
<h2>What is Content Strategy</h2>
<p>Content strategy is the practice and process of planning content creation, delivery, and governance. Its purpose is to create a repeatable system that defines the entire editorial content development process for a website.</p>
<p>Drupal.org is a very unique website. It serves many purposes:</p>
<ul>
<li><strong>Drupal.org is the home of our community.</strong> That makes different things to different people, but at its heart, Drupal.org is about the collaboration that allows us to build Drupal the software.</li>
<li><strong>Drupal.org is the canonical source for Drupal the software.</strong> Drupal.org binds together the respositories for Drupal core and contributed projects, issue queues for requesting features and reporting bugs, and packaging for automated building of releases that are tied to an integrated update process.</li>
<li><strong>Drupal.org is the hub of our commercial ecosystem.</strong> Companies that sell Drupal services and/or Drupal hosting are brought together with customers of Drupal the software—organizations using Drupal to power their websites.</li>
<li><strong>Drupal.org is a communication channel and it feeds other communication channels. </strong>We link to a lot of content on Drupal.org and the homepage gets lots of unique traffic.</li>
<li><strong>Drupal.org is a source of information.</strong> The site provides information about Drupal the software, Drupal.org the site, and the Drupal Association.</li>
<li><strong>Drupal.org is a place where people go to evaluate Drupal.</strong> Developers, Designers, CTOs, CIOs, and more go to Drupal.org to read about features and success stories to make a decision to use the Drupal the platform to build their content management solutions.</li>
<li><strong>Drupal.org is a starting point for support. </strong>Many users ask their first questions to the community using the Drupal forums or issue queues. The find answers by searching the Internet and being pointed back to the answer on Drupal.org.</li>
<li><strong>Drupal.org is a collection of documentation.</strong> Our canonical API documentation is generated from the repositories associated with Drupal.org. Our community has built pages upon page of documentation to help users understand how to build with Drupal and how to contribute to building Drupal.</li>
</ul>
<p>With so many purposes and competing objectives, a cohesive content strategy that takes in input from many contributors and users of Drupal.org is critical.</p>
<h2>Setting a Content Strategy Vision</h2>
<p>To keep us aligned, we outlined three major areas to keep measuring our work against: the big ideas, key messages, and our objectives for content on the site.</p>
<h3>Key Messages</h3>
<ul>
<li>Drupal.org is the home of Drupal and the Drupal community. It is the source of code, information and collaboration, which enables people all over the world to build flexible and scalable technology solutions together.</li>
<li>We are a global community of web practitioners—from project managers and writers, to designers and developers—contributing our unique skillsets to building and growing the adoption of the free and open source software that is Drupal.</li>
<li>Drupal is used by nonprofits, government, and Fortune 500 companies to architect customized, appropriate solutions for a wide array of organizational needs.</li>
</ul>
<h3>Content Strategy Objectives</h3>
<ul>
<li>Improve quality and findability of relevant content so that users can efficiently move through proficiency levels.</li>
<li>Reframe Drupal.org around all user roles and proficiencies so that all audiences are addressed.</li>
<li>Develop content governance for Drupal.org to improve the overall quality of content.</li>
<li>Improve user engagement within the Drupal.org community so that members form deeper relationships and become Drupal promoters and contributors.</li>
</ul>
<h2>Identifying Content Types and Gaps in our Content</h2>
<p><img src="/files/1.2-million-pieces.png" class="right" alt="1.2 million nodes, 2.4 million comments" />We have 17 active content types and over 1.2 million pieces of content on Drupal.org. (Really, this is just nodes, we have even more taxonomy terms and views that also represent displays of data.) That’s a lot of content. It’s more than 29,000 projects (modules, themes, distributions, etc.) and over 789,000 issues posted to those projects. We also have over 330,000 forum topics being discussed.</p>
<h3>The Curious Case of the Book</h3>
<p>With all of that content, 17 types does not quite give us the flexibility or degree of classification that we need to provide truly structured content. We have some content types that are used for so many different kinds of content that they’re virtually meaningless. We have over 12,000 nodes in our “book page” content type. Our book pages can be anything from documentation to landing pages to resource guides to topical pages to module comparisons… really we use them for just about everything.</p>
<p>During the content strategy project, we will explore ways to break our book pages into more meaningful content types that help new users find what they need.</p>
<h3>What’s in a Forum</h3>
<p><img src="/files/forum-post-authors.png" class="right" alt="92,029 users have created a forum post" />Another content type that gets used for more than it should is the forum topic. We use forums to post news, security announcements, discussions and even support requests. Yet at the same time, it is clear that forums are used far less now than several years ago. We had over 50,000 forum posts in 2008. We had only 11,000 in 2014.</p>
<p>For support and questions, our forums do not have comparable functionality to systems like <a href="http://drupal.stackexchange.com/">Drupal Answers</a>—powered by Stack Exchange. Many community members that provide support have already moved to that site to answer questions. Drupal.org is still a starting point for many newcomers to Drupal. One goal of the content strategy project is to make some decisions about where we can best direct newcomers for support.</p>
<h3>Where are the Marketing Materials to Help People Choose Drupal?</h3>
<p>A key classification of content that we are missing in our information architecture on Drupal.org is marketing materials. We create tons of documentation and handbooks, but we do not have a ton of great materials that tell business evaluators (CIOs, CTOs, managers, and decision makers) why they should choose Drupal. We have a good start with content created to <a href="https://www.drupal.org/drupal-8.0">promote Drupal 8</a>, but there is a lot more we can do to help sell the qualities of Drupal.</p>
<p>These are just a couple of the gaps that we have found and are working with the <a href="https://www.drupal.org/governance/drupalorg-working-groups/content">Drupal.org Content Working Group</a> and the <a href="https://www.drupal.org/governance/doc-working-group">Documentation Working Group to address.</a></p>
<h2>Auditing What We Have and Mapping What We Want</h2>
<p>We took the time to map our community’s content production over time and the totals were amazing.</p>
<p>The height of our community’s content creation was in 2012, when we created more than 195,000 nodes on Drupal.org and Drupal Groups. As Drupal 7 has matured, we have slowed down a bit. In 2014, we created 116,514 nodes on those two sites. That is still a huge amount of content.</p>
<p><img src="/files/percent-content-before-2011.png" class="right" alt="55% of Drupal.org book pages were created before Drupal 7. 32% have been updated since its launch in 2011." />Nearly 39% of all of the content on Drupal.org and Drupal Groups was created before 2010. More specifically, 55% of all book pages were created prior to the launch of Drupal 7 in 2011—that’s 5,665 book pages. Only 32% of those book pages have been updated since. That gap of 23% of all book content is a good place to begin an audit.</p>
<p>We are working now to finalize a process for identifying what content could be archived or removed and what content needs to be updated. The community has done admirable job of classifying our documentation by page status, but there is more work to be done. We need an automated process for regularly auditing our content.</p>
<p>We need a better map of related content—content we have and content we need—that can be used to build a better information architecture for new users.</p>
<p>One of the key deliverables for our content strategy project is a site map of what we want the site to look like in 3 months, 6 months and 1 year.</p>
<h2>Creating a Governance Plan to Better Support our Community of Creators</h2>
<p>We are hard at work reviewing and documenting community processes for maintaining content on Drupal.org. If users have been around for a while, they might have found their way into the <a href="https://www.drupal.org/project/content">content issue queue</a> and wondered at the process and how to start helping. <img src="/files/edits-to-books.png" class="right" alt="6,452 users have edited 12,326 book pages over 92,000 times." />They may also have jumped in and helped edit a documentation page in one of our numerous books. (6,452 of community members have edited 12,326 book pages over 92,000 times.)</p>
<p>The problem is that these processes are not well known and not built into our tools at a level that helps users know what they should and should not do in the system. Learning the “right way” to contribute requires finding policy documentation that is often difficult to get to, and sometimes out of date. Therefore, along with our new content types, we are assessing and testing the user experience for creating, curating and maintaining all of the content on Drupal.org.</p>
<p>As we document the existing rules that govern how contributions are made, it’s become clear that one of the greatest barriers to contribution, especially for new users, is the sheer difficulty of learning the “right way” to make a contribution. We want to change the way these users interact with the site, so that the correct process and procedure for each type of contribution is <strong>baked right into the workflow</strong>.</p>
<h2>Making our Communications Count</h2>
<p><img src="/files/communication-channels.png" class="right" alt="blogs, @drupal, Drupal.org, webcasts, Planet, Groups, newsletters, issues" />The last key deliverable that is being finalized as part of our content strategy is our communications plan. We have 50+ channels that are used by Drupal Association, working groups, social media volunteers, and maintainers to communicate with the community—everything from Twitter to newsletters to the Drupal.org homepage. We do not want to flood you with too much information, but we would like to be able to give you the information you want to see when you want to see it.</p>
<p>Right now, Drupal Association staff and the Drupal.org Content Working Group are mapping our messages to our audiences, our message to our channels and our channels to our audiences. It will be easier than ever to subscribe to the information you want—both email and on the site itself—in the coming year.</p>
<h2>Next Steps</h2>
<p>We will be wrapping up our content strategy work as March comes to a close.</p>
<p>We will publish more findings along the way. Stay tuned for new content types on Drupal.org—including news, posts, topic-based taxonomy term pages, and better ways to access and help write documentation.</p>
Strategic Initiatives for Drupal.org in 20152014-11-03T00:00:00+00:00http://joshuami.com/2014/11/03/strategic-initiatives-for-drupalorg-2015<p>Reposted from Drupal.org</p>
<p>Drupal.org is an amazing installation of Drupal. At nearly 13 years old, it is one of the largest, continuously operating examples of Drupal. It is difficult to fathom, but Drupal.org has been upgraded in place from version to version for this entire timespan. I can think of no other site that has gone this long without a significant content and structure migration.</p>
<p><a href="/ecosystem"><img src="https://docs.google.com/drawings/d/1Y7KkP0IgISsN8LuaAnJPKlxUxBe3883_oFS36It82vg/pub?w=300&h=177" class="right" title="Drupal.org ecosystem, click for larger view" /></a></p>
<p>Over the years, Drupal.org has grown from a single server owned by a contributor to multiple racks at the OSL data center, plus cloud resources and content distribution networks spread across the globe. Drupal.org is more than a single site. There are over 20 services and subsites that make up the ecosystem that powers the Drupal community. Each month, over 20 TB of data passes through the Drupal.org infrastructure.</p>
<p>With such a huge impact, it is important that we have a strong plan for the direction of Drupal.org. With that, we would like to introduce you to the <a href="/roadmap">Drupal.org Roadmap</a>.</p>
<p><a href="/roadmap" class="link-button"><span>Drupal.org Roadmap</span></a></p>
<p>Read on to find out how we set this strategic direction.</p>
<h2>History</h2>
<p>Volunteers built up these systems focusing on their passions with community initiatives. Many times these volunteers gave up days of their life - unpaid - to make sure that people could continue to build websites with Drupal and to build Drupal and its contributed projects.</p>
<p>While the result is impressive, there are many areas of Drupal.org that received little or no attention in this model of development. If a developer burned out, or there was no one in the community with a passion for the area of needed improvements, that area remained unmaintained.</p>
<p>For several years, the Drupal Association has funded the infrastructure that runs Drupal.org. The Association pays for the hosting facilities and the hardware to keep Drupal.org running.</p>
<h2>The evolving role of the Drupal Association</h2>
<p>In 2013, the Drupal Association board made the decision to begin building up an engineering team. This team would support both the infrastructure and software development activities behind Drupal.org. Our goal is to accelerate the development of the new features and to help build a cohesive roadmap so that Drupal.org would help unite a global community to build the best of the Web with Drupal. (Hint: that is the <a href="https://assoc.drupal.org/about">mission of the Drupal Association</a>.)</p>
<p>Hired in March of 2014, the Drupal Association CTO was tasked with building a team and gathering feedback from Working Groups and the Board of Directors to set a strategic direction for Drupal.org.</p>
<h2>Prioritizing the work</h2>
<p>There are three primary working groups that guide the development of Drupal.org: Drupal.org Content Working Group (DCWG), Drupal.org Software Working Group (DSWG) and Drupal.org Infrastructure Working Group (DIWG).</p>
<p>New development of features for the Drupal.org community of sites and services was determined through weeks of careful deliberation and research:</p>
<ul>
<li>Previous years of feature ideation</li>
<li>Working group feature ideation</li>
<li>User research project</li>
<li>Working group prioritization</li>
<li>Board of Directors input and feedback</li>
<li>Staff ideation on maintenance and performance improvements</li>
</ul>
<p>One of the key influences in our prioritization process was the user research that was conducted during and after DrupalCon Austin in June of 2014. We interviewed over 30 individuals that represented a wide range of Drupal.org users from those that were just starting with Drupal, to longtime members of the community, and even those that had once used Drupal and had transitioned their careers to different technologies.</p>
<p>This gave us four key areas in which to focus:</p>
<h2>Sustaining support and maintenance</h2>
<p>These efforts are the ongoing work that keeps the servers up and running and performing well. The <a href="/project/infrastructure">Drupal.org Infrastructure issue queue</a> is the primary place for this work, but there are several other related queues where staff and volunteers from the infrastructure team are focusing their work. Work that staff is tackling will be assigned to a staff member and tagged with d.o support.</p>
<ul>
<li>Support for users: Drupal.org issue queues and email support</li>
<li>Performance: uptime, page response, ongoing testbot deployments and maintenance</li>
<li>Improving automated tests to make development and deployment reliable</li>
<li>Maximize hardware and migrate to cloud services where appropriate</li>
</ul>
<h2>Fund Drupal.org and future tools</h2>
<p>While the majority of funds supporting Drupal.org come from our partner programs (Supporting Partner, Technology Partner, Hosting Partner), we are looking for ways to diversify where we raise funds.</p>
<ul>
<li><a href="https://jobs.drupal.org/">Drupal Jobs</a></li>
<li>DrupalCon websites (<a href="https://latinamerica2015.drupal.org/">Latin America</a>, <a href="https://losangeles2015.drupal.org/">Los Angeles</a>, <a href="https://barcelona2015.drupal.org/">Barcelona</a>, etc.)</li>
<li><a href="http://drupalstore.org/">Drupal Store</a></li>
<li><a href="https://assoc.drupal.org/advertising/web">Drupal.org advertising and partner program improvements</a></li>
<li>Tools to help promote and add value to a <a href="https://assoc.drupal.org/membership">Drupal Association membership</a></li>
</ul>
<h2>Board and Working Group Priorities: Drupal.org Staff Initiatives</h2>
<p>These initiatives represent the work that Drupal Association technology and engineering staff will be focused on in the near term through 2015. By being focused on these initiatives, we will get the related features launched on Drupal.org faster. We will still need help to vet and test these features, so follow the issue tags you are interested in and get involved in the related issues.</p>
<ul>
<li>Better account creation and login </li>
<li>Organization and user profile improvements</li>
<li>Responsive Redesign of Drupal.org</li>
<li>Issue workflow and Git improvements</li>
<li>Make Drupal.org Search Usable</li>
<li>Improved tools to find and select projects</li>
<li>Groups migration to Drupal 7</li>
</ul>
<p>The <a href="/roadmap">Drupal.org Roadmap</a> provides much more detail about these key initiatives.</p>
<h2>Community Initiatives</h2>
<p>There is always more work to do on Drupal.org. We need committed and active volunteers to help with key initiatives that showed up in both our user research and the prioritization from the working groups. These are projects that we can support the efforts of contributors that have the time and skills to push these initiatives forward. Three examples with strong community leadership include:</p>
<ul>
<li>Support localize.drupal.org</li>
<li>Next generation testbots (DrupalCI)</li>
<li>Two-factor authentication</li>
</ul>
<h2>You can help</h2>
<p>In addition to these initiatives, we would love to support a community member that would be willing to step up and lead an initiative to organize our Q&A and support on Drupal.org. There is a huge need for people to be able to find answers to their Drupal questions. Stack Overflow fills part of this role, but there are many more opportunities on Drupal.org itself.</p>
<p>We will also need a community driven effort to help us establish project ratings and reviews once these tools are in place. It will take a group effort to make these affective quickly.</p>
<p>All of our initiatives need community involvement. Whether it is commenting on issues posted to these projects or joining in at sprints to move these initiatives forward, we can use your time and commitment.</p>
<h2>Thank you</h2>
<p>This planning and work would not have been possible without the financial support from our partners, the direction and leadership of the board, the time commitment of our Working Group members, and an incredibly dedicated Drupal Association staff.</p>
<p>Cheers!</p>
Drupal.org is staffing up2014-04-30T00:00:00+00:00http://joshuami.com/2014/04/30/drupalorg-is-staffing-up<p><a href="https://assoc.drupal.org/blog/joshuami/drupalorg-staffing">Reposted from Drupal Association website</a></p>
<p>The Drupal Association Board of Directors has committed to significantly improving Drupal.org in 2014. As a result, we are staffing up, working with contractors, and improving volunteer engagement paths so that we can really accelerate change on the site this year and keep the momentum going into next year.</p>
<h3>
What’s next for Drupal.org?</h3>
<p>The primary imperative outlined in our 2014 Leadership Plan and Budget is to make Drupal.org shine. Here are the things I am going to be focused on most immediately to that end:</p>
<ul>
<li>
Working Groups: Work with the Software, Infrastructure, and Content Working Groups to quickly get up to speed and further clarify processes and systems so we can accelerate decision making for Drupal.org</li>
<li>
Build a strong internal team at the Association and put systems and process in place to support a more transparent and routine deployment process.</li>
<li>
There is a fair amount of technical debt to pay off. We’ll be focusing on two types of projects:
<ul>
<li>
removing potential blockers to a Drupal 8 release (i.e. improving testbot performance and implementing semantic versioning in our git repositories)</li>
<li>
accelerating volunteer involvement</li>
</ul>
</li>
</ul>
<h3>
We are adding infrastructure, engineering and design staff</h3>
<p>Through most of 2013, we had two technical positions on the Drupal Association staff. This was not enough to do much more that keep up with pressing issues in the issue queue and support the volunteers put in massive work in maintaining our infrastructure. In 2014, we are hiring the following positions:</p>
<ul>
<li>
CTO of Drupal.org (that’s me)</li>
<li>
Drupal.org Infrastructure Manager (filled)</li>
<li>
<a href="https://assoc.drupal.org/jobs#op-30612-drupalorg-devops-engineer">Drupal.org DevOps Engineer</a> (currently posted)</li>
<li>
<a href="https://assoc.drupal.org/jobs#op-33504-drupalorg-securityarchitecture-engineer">Drupal.org Security/Architecture Engineer</a> (currently posted)</li>
<li>
<a href="https://assoc.drupal.org/jobs#op-33505-drupalorg-qa-engineer">Drupal.org QA Engineer</a> (currently posted)</li>
</ul>
<p>We also need design (theming) and UX support to help push us forward. To that end, we are posting three positions with the intent to get the best possible single hire for 2014.</p>
<ul>
<li>
<a href="https://assoc.drupal.org/jobs#op-33499-drupalorg-product-manager">Drupal.org Product Manager</a> (currently posted)</li>
<li>
<a href="https://assoc.drupal.org/jobs#op-33503-drupalorg-web-designer">Drupal.org Web Designer</a> (currently posted)</li>
<li>
<a href="https://assoc.drupal.org/jobs#op-33501-drupalorg-user-experience-engineer">Drupal.org User Experience Engineer</a> (currently posted)</li>
</ul>
<p>Additional design and user experience positions are planned for 2015. We will continue shore up our technical team with committed volunteers and will also be looking to strategically use contractors to move important projects forward.</p>
<h3>
Help us find the best</h3>
<p>Do you know someone that you think would be a great addition to the Drupal team? Ask them to apply. More information can be found at <a href="https://assoc.drupal.org/jobs">https://assoc.drupal.org/jobs</a>.</p>
Drupal.org Response to Heartbleed Security Incident2014-04-08T00:00:00+00:00http://joshuami.com/2014/04/08/drupalorg-response-to-heartbleed<p><a href="https://www.drupal.org/news/2014-04-08-security-update">Reposted from Drupal.org</a></p>
<p>You may have heard that a vulnerability in the OpenSSL cryptographic library called Heartbleed or formally called CVE-2014-0160 has been disclosed and that it represents a potential security threat to a large number of websites. Using this vulnerability, malicious individuals could access sensitive information submitted by people actively visiting a website including usernames, passwords and credit card numbers. Users across the Internet should be especially aware of suspicious activity on their accounts.</p>
<p>We want to communicate a couple pieces of information about this news with regard to Drupal.org.</p>
<p>Members of the Drupal Association staff, Drupal Security Team and Drupal Infrastructure Team have reviewed Drupal.org’s potential exposure to the vulnerability.</p>
<p>As of now, we have no indication that Drupal.org was attacked using this vulnerabililty. That said, the nature of the vulnerability makes an attack difficult to detect and we prefer to be cautious.</p>
<p>We have taken steps to protect users of Drupal.org, including a forced password reset for users with administrative access or access to code repositories for projects. While we have only forced the password reset for some users, we recommend that all of our users change their passwords.</p>
<p>We have taken the following steps to protect Drupal.org account holders:</p>
<ul>
<li>Installed new SSL certificates based on a new private key</li>
<li>Revoked the old SSL certificates</li>
<li>Replaced the private strings (drupal_private_key and drupal_hash_salt) which are used for a variety of security related purposes in all Drupal sites</li>
<li>Replaced the private key used by the “bakery” single-sign-on system on Drupal.org</li>
<li>Removed all active sessions</li>
<li>Verified the email addresses in use today match those in use a week ago</li>
<li><strong>Required that all Drupal.org users with administrative or project repository access to reset their passwords</strong></li>
</ul>
<p>Also, we simply want to help create awareness about the vulnerability and encourage people to review their sites for exposure. For more information, please see https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-0160</p>
<p>Feel free to comment on the post with any questions. Thank you!</p>
Drupal at Multnomah County2013-01-08T00:00:00+00:00http://joshuami.com/2013/01/08/drupal-at-multnomah-county<h2>Executive Buy-in</h2>
<p>In October of 2009, Multnomah County hired their first position dedicated to the development of open source technology and solutions. (That would be me.) This mid-level management position was tasked with identifying and vetting open source software, hiring open source developers and managing IT applications for the general government line of business.</p>
<p>General government is a broad term that includes departments like County Management (which includes County HR, finance, payroll, benefits, assessment, recording and taxation, etc.) , Community Services (wich includes animal services, bridge maintenance, road maintenance, elections, and land use planning), elected officials, the county attorney’s office and, to top it off and round things out, the public and intranet website platform for all departments. Similar development groups exist for the health, human services and public safety related departments.</p>
<p>Three areas were identified as a starting point for the use of open source technology: digital document management, rapid application development and blogging for the Multnomah County Library. General content management for the public website was considered off the table as a redesign project that included an upgrade to the latest version of Vignette Content Portal was already underway.</p>
<p>Drupal was initially identified as the platform for rapid application development and content management for internal solutions such as Intranets and collaborative workspaces. At that time, we could use Drupal as a CMS, just not as <em>the</em> CMS as we had invested so much into Vignette.</p>
<h2 dir="ltr">A Change in Direction</h2>
<p>A year into the website redesign, it became increasingly clear that while we had optimized our processes for migrating from Vignette 7.3 to Vignette 7.5 (if you are a Drupal user, think Drupal 5 to Drupal 6—same concepts, but big changes), we were not able to hit the pace necessary to complete a site-wide redesign by our target date of February 2011. We had given ourselves two years for the project and revised it to three years because of the sheer scope of moving every individual site that was either Vignette or static HTML into this system. At the time I considered this an insane amount of time. I was used to rolling projects in months (if not occasionally weeks)… not years.</p>
<p>Of a greater concern was the difficulty we had in training our end users to edit their Vignette content. While Vignette scales well to heavy traffic and has a solid enterprise reputation, it is not an intuitive system for non-technical end users. Vignette lacks a strong rich text editor and image handling out of the box and its system of “projects†and “channels†is simply not a lexicon that is easily picked up by end users.</p>
<p>In mid-April 2010, myself, the County web manager and the project manager for this project put together a convincing argument to shift our migration efforts from Vignette to Drupal. I took two developers to DrupalCon in San Fransisco—nicely timed—for a crash course in Drupal development. I wanted to get my two leads up to speed on the latest trends in configuring a large-scale Drupal website as quickly as possible. At the time, I was the only one in the group with experience with Drupal.</p>
<p>A target for an initial pilot was set for July 1st to launch two sites: public affairs and our jobs site that HR maintained. Fortunately, I was able to hire an open source developer with tons of Drupal experience in June, but the pilot that launched received only a cursory review from him before we were schedule to go live.</p>
<p>We hit our target launch date with a fully-functional, highly-scalable website that served up for 65,000 pageviews in the first month.</p>
<p>Upon completion of a successful and on-schedule pilot, the redesign effort was completely shifted to Drupal. As of January 3rd, 2011, roughly six months from the completion of the pilot, 45 sites were launched on the new Drupal platform. 16 sites remain; eight of which are are in progress. The effort from each site launch has moved from web development and engineering to content migration. Content migration is completed primarily by departmental editors assisted by a team of two part-time interns, a part-time training specialist and the county web manager (responsible for helping departments restructure their content and planning site publication schedules).</p>
<p>A new “site” within the County’s Drupal installation takes approximately two minutes to create and publish. A typical site migration takes less than a week—including site reorganization.</p>
<h2 dir="ltr">The Platform Behind the Progress</h2>
<p>Multnomah County’s Drupal platform is built on Amazon’s Elastic Computing environment. The EC2 image in use is preconfigured with Pressflow (an optimized version of Drupal that takes advantage of advances in MySQL and PHP) with special configuration to take advantage of packages such as Varnish for HTTP caching, Memcached for MySQL caching, and APC for PHP caching.</p>
<p>The development environment follows best practices using Subversion <em>[note: we moved to git in 2012]</em> for code revisioning in a four stage deployment process to allow for localhost development, a quality assurance server, a user acceptance testing server and the public production server. The database backed up automatically every two hours. Twice a day, production backups are copied between all environments. The production and UAT environments are hosted out of Amazon’s Western zone with elastic IPs that allow for manual fail over between environments in five seconds or less without waiting for DNS propogation. The testing environment is hosted out of Amazon’s Eastern zone for disaster recovery fail over that is geographically isolated.</p>
The total cost of your government project... easy... right?2011-05-16T00:00:00+00:00http://joshuami.com/2011/05/16/total-cost-government-project<p>I had an interesting conversation with a fellow County employee today about the importance of figuring out total cost of ownership and return on investment.</p>
<p>Neither of these topics are particularly interesting in and of themselves, but in the current economic times they make all the difference in whether a government project will be funded.</p>
<p>First, there is the question of total cost of ownership. What does it take to get to this number? Many people would stop at the cost of licensing—especially if they have an old server lying around and the software in question is open source. After all, open source is free—right?</p>
<p>The notion of free software sounds great, but there is more to it than that as any proponent of open source software will tell you. Software takes support. Paid employees provide that support. So when figuring out cost of ownership, every manager should take a moment to figure out the cost of support.</p>
<p>Automation through technology is a wonderful thing, but while technology may drive down the costs of tasks that were formerly completed by people, the implementation and support of that technology will usually mean an increase in IT costs. Factoring IT costs into the cost of ownership is critical to knowing just how much you’ll spend over the life of the technology implemented.</p>
<p>Total costs for a typical open source IT project include infrastructure, implementation and support. Infrastructure is the hard costs of the servers, storage and power used by the software. Implementation is the people time it takes to get the solution in place—it sometimes takes years to pay off that initial cost if the software is difficult to implement or complex. Support is the time it takes to maintain a piece of software. Support starts out high, decreases as software becomes familiar to its users, and then increases as time passes and the software gets closer to retirement.</p>
<p>So, if you think that six year old piece of hardware slated for retirement is going to save you money—you might want to think again. That server is going to cost more to support. It will crash more; it will have more security issues; it will use more power. Just like our health care system, older things cost more to maintain and keep alive. Though there is something to be said for not having to spend the time in purchasing a new server—and no, my health metaphor doesn’t extend this far.</p>
<p>Another option to consider would be a software as a service solution (SaaS). SaaS solutions have the advantage of a larger customer base. The more customers to which you provide a service, the lower the cost. This is why a server at Amazon Web Services can cost so little per hour of use. Lots of server admins are using that service and driving down the costs of storage and power, which Amazon can pass on to us. SaaS can take the same savings a level deeper and provide a lower cost for providing a service that could be internally provided because of the scale of service.</p>
<p>What about return on investment? The project I was discussing with my colleague today was a chat server that ostensibly would reduce the call volume for his department. That same software would increase the options for citizens to use his service. It might have the reverse affect of increasing traffic because people would be more wiling to chat with an “operator†than to call them. In this latter scenario, it might mean increased costs for the investment—a negative ROI.</p>
<p>Would that negative ROI be acceptable if the public was happy with the increased service? Tough call.</p>
<p>In the end, I think this project shows a lot of promise. It really would have the potential for reducing call volume and increasing the number of citizens served by the same number of government employees. It is that sort of innovation that we need at times like these. We also need to be mindful of the total cost of innovation and what we are getting in return.</p>
I'm hoping for the dragons2008-04-02T00:00:00+00:00http://joshuami.com/2008/04/02/hoping-for-dragons<p>According to <a href="http://www.treehugger.com/files/2008/03/end-of-the-world-hadron.php">some wacky scientists</a> out there, there is a possibility that turning on a really powerful nuclear device could have unexpected consequences. Pbbwfff!</p>
What Would Boxes and Arrows Do? (WWBAD)2008-02-24T00:00:00+00:00http://joshuami.com/2008/02/24/what-would-boxes-and-arrows-do<p>In the course of handing off a project to another company, I just had the opportunity to say “just build them <a href="http://www.boxesandarrows.com/">Boxes and Arrows</a> on <a href="https://www.drupal.org/">Drupal</a> and they’ll be thrilled.” While I was merely using hyperbole to make a point, it got be thinking. What do they use to deliver their content?</p>
<p>The answer is <a href="http://publicsquarehq.com">PublicSquare</a>. This is a subscription-based service for creating an online magazine with a community component. I’ve always like the layout and general functionality of Boxes and Arrows and now I know what they use to serve up their content.</p>
<p>I plan to look into this a bit more as the idea of a subscription-based magazine CMS intruiges me. I would have signed up for an account today, but it turns out that they have an expired security certificate on their account sign up page. I wonder if they know that? (It expired on December 6, 2007.)</p>
Competition in the online community software market2007-11-13T00:00:00+00:00http://joshuami.com/2007/11/13/competition-online-community-software-market<p>I just took a look the features and FAQ for the new <a href="http://www.movabletype.com/products/community-solution.html">Movable Type Community Solution</a></p>
<p>The community solution boasts the following:</p>
<ul>
<li>Member registration with extended user profiles</li>
<li>Forums</li>
<li>Community blogs</li>
<li>Custom fields</li>
<li>Recommendation engine</li>
<li>Sort-by-popularity</li>
</ul>
<p>The feature set overlaps heavily with [Clearspace X}(http://www.jivesoftware.com/products/clearspace/clearspacex.jsp) from Jive Software. I’ve seen Clearspace up close, and I think it is a great product, but a community site based on the code of Movable Type with its extensive plugin architecture and significant user base could be a significant competitor. Equally compelling for smaller organizations would the combination of WordPress and BBPress as both are completely opensource—free as in beer, but arguably less stable due to the more rapid release cycle. The growth of solutions in this space has been rapid. There seems to be plenty of room in the blog/community solution space for more players. It should be an interesting market to watch grow over the next couple of years.</p>
A busy weekend2007-10-22T00:00:00+00:00http://joshuami.com/2007/10/22/busy-weekend<p>So the weekend started off with a leisurely Saturday morning—we slept in until 7:15, woohoo—but it quickly degraded.</p>
<p>First there was the startled cry from Diana to inform me that the wall in our daughters bedroom was leaking. We had some pretty hard rains over the last few days leading up to the weekend. Saturday was the culmination of that storm, but the weakening of the wall started a long time ago.</p>
<p>The previous owners thought it would be nice to replace the aluminum windows in the back of the house with vinyl. A wise decision, but their method left me with a mess. They never removed the old aluminum frames. The just took out the glass, cut a couple of pieces of the frame off, and installed the vinyl up against the aluminum. The vinyl window frame was actually sitting against plaster on the top and sides. Plaster is not a material that can be sealed so there was some mold growth in those areas.</p>
<p>I am sure this installation method saved the previous owners money.</p>
<p>The result of this shoddy installation was a small lip where water could collect between the aluminum and vinyl frames. Water has some amazing physical properties—it will find weakness and penetrate it.</p>
<p>Over time water was slowly seeping into the cracks between the two frames and down into the outer wall. It was also finding its way into the inner wall through lathe and plaster about two inches to the right and four inches down from the right corner of the window. The result was a patch of mushy plaster that we had noticed last winter—though at the time we thought it was being caused from condensation and a small leak that some silicon in the spring was to have fixed.</p>
<p>So what’s the damage? Well, I managed to put up a tarp to stop the rain from continuing to soak the frame around the window. I then proceeded to remove the window and inspect the damage. The studs and bottom frame are in surprisingly good shape. The plywood exterior behind my aluminum siding is not. It was completely rotted out in a two foot patch. I only know this because I removed enough of the drywall to inspect from the inside. (Yes, I can see the back of my siding from inside my house right now.) I was able to remove enough plaster to rebuild the frame with some filler 3/4” stock and reinstall the window.So I don’t have a whole in my house–for now. The next step is going to involve removing the siding from the back corner of the house and replacing the plywood exterior and rewrapping it in Tyvek.</p>
<p>So do you think this was all I did this weekend? Nope, it is not even close. We also fit in pumpkin carving, buying my wife a new phone (the Samsung Juke, more to follow on that), getting some desperately needed new shoes (Keens of course), a frantic call from a client (freelance gig) to set me off onto a search for why my host decided to change DNS for a mirror domain, and my first ever complete installation of Drupal.</p>
<p>Yeah, what did you do this weekend?</p>
From the mouths of babes2007-10-21T00:00:00+00:00http://joshuami.com/2007/10/21/from-the-mouths-of-babes<p>Shya says, “the human floss tastes funny on my tongue.”</p>
Cyldesdales coming to the Cross Crusade2007-10-19T00:00:00+00:00http://joshuami.com/2007/10/19/clydesdales-coming-to-cross-crusade<p>[Via Bike Portland] <a href="http://bikeportland.org/2007/10/18/cyldesdales-coming-to-the-cross-crusade/">Cyldesdales coming to the Cross Crusade</a>. I qualify for this category. I wonder if there is a cyclocross bike loaner program for us beefy riders? That’s a lot of extra stress on the bike!</p>
<p>Note to anyone concerned for my health: I weigh considerably less than when I was a offensive lineman on my high school football team when I weighed in excess of 260 pounds. My svelter self is in large part due to cycling and the calories I burn commuting to work and riding on the weekends. With my 6’ 3” frame, I doubt I could ever weigh less than 200 pounds. That said, I could stand to lose about 10 pounds right now as my commute-by-bike options have been hampered by the kids’ new school schedule.</p>
<p>A couple clydesdale races might be just the ticket for burning off some calories and raising the endorphine levels a bit.</p>
But what CAN they eat?2007-10-08T00:00:00+00:00http://joshuami.com/2007/10/08/but-what-can-they-eat<p>A concerned parent intending to bring snacks to my son’s third-grade class wanted to make sure the food she brought would be appropriate for everyone in the class, so she put together a list of food that were acceptable:</p>
<blockquote>These are all vegetarian, wheat free, dairy free, soy free (all oil will specifically be canola to avoid soy), free of artificial flavors, synthetic colors, BHA, BHT, TBHQ, and as far as I can tell, free of citric acid (I'm not sure what all is in this category—citrus fruits, berries, pineapple, tamarind, cherries, tomatoes, cayenne, artichokes, lettuce, cheese (ruled out by dairy anyway), and sourdough).</blockquote>
<p>So what CAN they eat? I was surprised to see a list of five options:</p>
<ul>
<li>Frozen grapes & pistachios</li>
<li>Banana bread with peanut butter</li>
<li>Pumpkin muffins with almond butter</li>
<li>Greek lentil soup with cornbread muffins made with rice milk</li>
<li>Steel cut oats with vanilla, maple syrup and toasted pecan</li>
</ul>
<p>…so apparently we don’t have any nut allergies. Does anyone else find this list of allergies peculiar?</p>
Made it to school by bike again... nearly a month later2007-10-05T00:00:00+00:00http://joshuami.com/2007/10/05/made-it-to-school-by-bike<p>So it took nearly a month to make it to school by bike again. Nearly killing your child by not attaching the trail-a-bike correctly can do things like that to your schedule.</p>
<p>Someday I really will post pictures of our biking arrangement.</p>
Terms in WordPress 2.32007-09-25T00:00:00+00:00http://joshuami.com/2007/09/25/terms-wordpress-2-3<p>The WordPress 2.3 terms table is a smart, forward-thinking update… and it breaks most themes horribly.</p>
First Look at Mint2007-09-20T00:00:00+00:00http://joshuami.com/2007/09/20/first-look-mint<p>I just took a first look at <a href="http://www.mint.com/">Mint</a>, a new financial management tool that just won a place in the <a href="http://www.techcrunch40.com/2007/index.php">TechCrunch40</a>. I am impressed. I’ll follow up with a more in depth review once I’ve worked up the nerve to share my online banking information with them…that’s a big leap for me. Until then, here are a few salient points:</p>
<ul>
<li>Fast. Signup is quick and easy and the tool itself it lightning quick.</li>
<li>Free. They fund themselves by presenting you with offers from banks and credit card companies who can beat our your current interest rate. You don't have to accept any of these offers to take advantage of the free account.</li>
<li>Easy to use. Mint downloads all of your transactions through your banks online services. (The list of banks is impressive. All three banks I use for checking, savings or credit are in their database.) Once transactions are downloaded, their algorithm goes to town figuring out how you spent your money and giving you shiny reports and pie charts. Mmm, pie charts.</li>
<li>Claims to be secure. Sorry, this is something I am hesitant on. They appear legit in every way. The banks trust them. They are <a href="http://www.truste.org/">TRUSTe</a> certified and they use a third party service called <a href="http://www.yodlee.com/">Yodlee</a> to verify your account credentials so they don't have to store them on their servers.</li>
</ul>
<p>So the only thing that really scares me about this service is how much they will know about you in such a short period of time. They will know where you live and exactly where you spend your money. This is even scarier than <a href="http://en.wikipedia.org/wiki/RFID">RFID</a> in some ways… but it is also the one thing I can’t stand sinking time into on a weekly basis.</p>
<p>I have been using either Quicken or <a href="http://moneydance.com/">Moneydance</a> for about 12 years now. I rely on that software to tell me how well or how poorly I’m doing financially at any given moment. However, it is a lot of work and a constant headache. Especially since I have to manually download a <a href="http://en.wikipedia.org/wiki/QIF">QIF</a> from one of my financial institutions and “import” it to get transactions. Such a <a href="http://en.wikipedia.org/wiki/PITA#Abbreviations">PITA</a>.</p>
<p>If I get up the nerve to go deeper into this, I will post screenshots. <em>(No, I will not include my transaction info.)</em></p>
Google "Presentation" in the Wild2007-09-18T00:00:00+00:00http://joshuami.com/2007/09/18/google-presentation<p><a href="http://googleblog.blogspot.com/2007/09/our-feature-presentation.html">Google adds Presentation to Google Docs</a>. Interesting tool. I particularly like the feature that lets you publish the document and share a chat room and the presentation at once. Pretty quick way to set up a meeting. That said, I’m probably not giving up <a href="http://www.apple.com/keynote/">Keynote</a> for it.</p>
Campaign Monitor strikes a chord.2007-09-18T00:00:00+00:00http://joshuami.com/2007/09/18/campaign-monitor-strikes-a-chord<p><a href="http://campaignmonitor.cmail2.com/e/247188/xt64iir/">Campaign Monitor’s September 2007 Newsletter</a> is packed with great stuff. Besides the design gallery, which is always a treat, the newsletter highlights <a href="http://www.campaignmonitor.com/blog/archives/2007/09/why_we_need_web_standards_supp_1.html">David Griener’s treatise on the state of email standards</a> (great read) and throws out an article on <a href="http://www.campaignmonitor.com/blog/archives/2007/08/improve_your_deliverability_wi_1.html">email authentication</a>–which is so very timely. Don’t get me started on the new suppression list feature–handy–and the new women’s t-shirts! (Okay, so I’m not really in need of the women’s t-shirts, but I’m sure they make someone happy.)</p>
Woohoo! Biking to school and loving it2007-09-07T00:00:00+00:00http://joshuami.com/2007/09/07/biking-to-school-and-loving-it<p>Yesterday represents the first time that my kids and I biked to school. As a long time advocate of biking whenever possible, this was really cool to get my 8 and 6 year olds out in the fast lane–well, 8 mph is not exactly fast.<!--more--></p>
<p>The skinny:</p>
<ul>
<li>1 mountain bike for Dad (Gary Fisher Marlin 1999)</li>
<li>1 trail-a-bike for the little kid (Adams Trail-a-Bike)</li>
<li>1 mountain bike for the big kid (Raleigh Rowdy)</li>
<li>~30 minutes of ride time</li>
<li>~9 mph average speed</li>
<li>4.8 miles with kids</li>
<li>5 miles to work without kids</li>
</ul>
<p>(Pictures to follow, I forgot to take one in the rush to get out the door.)
Keeton did awesome on the bike lanes of North Portland. The Columbia Blvd. trail leaves a little to be desired at a couple of spots, but it was safe and traveled well. Mishya is now officially over her fears of biking. (I think the trailer was just a little too much loss of control as of late–the trail-a-bike is a much better fit for a 6-year old.)</p>
<p>My only complaints are that the coupling on the trail-a-bike was having a squeak-a-thon–it has been checked, it is not part of the recall from a year ago–and that the mountain bike is way heavy compared to my usual aluminum road steed. If I do this regularly, which is the plan, I think I’m going to need to switch to some lighter tires. I love the glass-crunching-ride-with-abandon fell of the Specialized Armadillo Crossroads I currently have on the Marlin, but man the tires feel heavy. I have to find something that will give me the top end speed of the slicks I used to ride with the puncture protection. I can then save the crossroads for trail riding–which I don’t get to do that often.</p>
<p>Two final issues with riding bikes to school that is 4+ miles from home. I can’t do pick up, so Diana had to swing by and grab the bikes with a car in the afternoon and Keeton and I crossed-wires regarding his helmet. I hope the helmet is sitting at school, but if not we are going to need to get him a new one before we can ride again. Darn.</p>
FreshBooks is hiring again2007-09-07T00:00:00+00:00http://joshuami.com/2007/09/07/FreshBooks is hiring again<p><a href="http://www.freshbooks.com/blog/2007/09/07/watch-out-film-fest-here-comes-freshbooks/">FreshBooks is hiring again</a>.</p>
<p>This is quite possibly the most inventive call for applicants I have ever seen. Proof that with a burgeoning company and a copy of iMovie you can make yourself look like fools, but still come off in a positive light.</p>
<p>Nice job FreshBooks.</p>
<p><em>Disclaimer: I use and love this product.</em></p>
Redefining the "inner" city school2007-09-05T00:00:00+00:00http://joshuami.com/2007/09/05/redefining-the-inner-city-school<p>There are schools in poor neighborhoods that are typically called “inner city schools.” These schools, we are told, harbor the most difficult students. These are schools that are euphemistically called diverse–meaning “not white”–the new melting pot. These are the schools most likely to struggle to meet the federal and state mandates for adequate yearly progress–or whatever the buzz word for “failing” is this year. These are the schools that have the least resources per student because of aging buildings and an indifferent tax base.</p>
<p>Only one thing stands out as a challenge to our presumptions, the schools are no longer located in the “inner” city.</p>
<p>These schools lie on the outer edge of the city–an endless flat expanse of one story buildings, strip malls, and mid-to-late century ranch homes. This is the new “outer” city school. Outside the boundaries of urban renewal and well inside the boundaries of urban growth are neighborhoods that are forgotten when bond measures are needed for revitalizing school buildings and restoring community centers.</p>
<p>The inner city schools in Portland, Oregon have begun the arduous process of revitalization through specialization. Nearly every school is a magnet school, or a charter school, or a focus area school. Interested in the Arts? Go to Da Vinci or Buckman. Interested in technology? You can sign up at Benson or George. Want an international education? Perhaps you will find Franklin’s international baccalaureate of interest or Grant’s Japanese program. Care about the environment? Sunnyside’s environmental program will nurture that aspect of your child.</p>
<p>As a parent in Portland, you hear about all of these programs. Many of them cause you angst and no small amount of jealousy at the unequal treatment some children are able to receive based on the luck of the lottery or the convenient–if somewhat more costly–location of their home.</p>
<p>What you don’t hear about as a Portland parent is the outer city schools and their programs. David Douglas is in Portland, but not the Portland Public System. So is Centennial. Both of these schools lie in the far away land called “East Portland” or “the East side.” The part of Multnomah county that doesn’t seem to want to vote with the rest of the urbanites come election time. The red menace in a decidedly blue landscape.</p>
<p>Why would they vote for more money in schools? Do they see this money flowing into their neighborhoods? Is there revitalization to be had in an area so far from the nearest freeway and steady middle class jobs?</p>
Configure versus code2007-01-18T00:00:00+00:00http://joshuami.com/2007/01/18/configure-versus-code<p>I’ve been running into this conundrum a lot lately. What makes something a configuration task versus a coding (or programming if you like) task?</p>
<p>I’ve come up with a series of equations that I think explain the issue:</p>
<p>XML == config file
Apache != config file</p>
<p>GUI == configuration task
text file != configuration task</p>
<p>settings == configuration
syntax == programming</p>
<p>praise == configuration
profanity == programming or a really bad UI</p>
<p>So it comes down to this. If you want to produce software for people to use, make it configurable. I’m tired of running into software that has no discernible administrative features–just a giant text file that is titled “config” or something similar.</p>
<p>I recently used Wordpress to help a friend launch a portfolio site. It ended up being the site’s CMS (Content Management System). Wordpress is not the best CMS available, but it is a simple solution that has a well-developed configuration model. That is not to say that I didn’t end up programming a bit in PHP–I certainly got my hands dirty. About half of what I did in the end was configuration and the other half was coding.</p>
Things to know when updating a Wordpress one-click install2006-11-12T00:00:00+00:00http://joshuami.com/2006/11/12/wordpress-one-click-install<p>I just finished updating the theme for Portland Tillamook Cooperative Preschool’s Web site using the Dreamhost one-click installer. (Well, it was their one-click updater really.)</p>
<p>Two very valuable lessons were learned.</p>
<p>One, Dreamhost likes to overwrite any existing themes with the same names (directories) as the themes installed with their default. This means anyone who has simply edited a default theme will lose their edits.</p>
<p>The second lesson I learned is that their default backup of the site to a directory with “.old” at the end is a very handy failsafe. Oddly enough, I did not have a local copy saved before clicking “upgrade.” A rare data-related mistake that I will not repeat.
On a final note, I certainly appreciate that Dreamhost installs some rather interesting themes with their one-click installer, but I wish it weren’t so many. It really slows down the page load on presentations. It also takes a while to delete them all.</p>
Turning over a new leaf... again... and again...2006-11-11T00:00:00+00:00http://joshuami.com/2006/11/11/turning-over-new-leaf-again<p>By far the most difficult aspect of blogging is keeping up the momentum of writing. The most successful bloggers may appear to have a ton of time on their hands–and many probably do–but they have something else as well. They have developed a drive to write and write often. (It probably helps when you suddenly start getting regular comments on your blog and are introduced to a taste of fame.)<!--more--></p>
<p>MobileHippie has languished for an extremely long time. I’ve never really been able to convince myself that blogging was important enough to commit to it. I certainly haven’t had the time to put any real effort into establishing a line of thought on which this blog can focus.</p>
<p>At first I thought a blog about being a designer/developer would be interesting, but I don’t design or develop enough these days to add anything meaningful to that conversation. I could write about being a project manager, but there are others with far more experience that have contributed outstanding articles covering effective project management techniques. I could write a more personal blog about being a parent or the state of public schools or the challenges and joys of raising a family in North Portland. I could write a bit about commuting by bicycle and the all-weather joy of being carless by choice–mostly.</p>
<p>In the end, none of these topics seem compelling enough to focus a regular blog. While there is little doubt that many of these topics will come up when I begin to write more regularly, they will not be the focus.</p>
<p>So what will be the focus? Coming soon.</p>
Two switches in one weekend2006-06-18T00:00:00+00:00http://joshuami.com/2006/06/18/two-switches-in-one-weekend<p>Along with the switch to Wordpress, I have also moved MobileHippie.com to my Dreamhost account. UplinkEarth still has a couple of my sites, but I’m consciously making an effort to move anything that uses Wordpress to a Linux server running Apache so that I get the full power of the software.</p>
<p>IIS and Wordpress were not meant for each other. I’m not really sure why ULE decided to make Wordpress a one-click install on their shared Windows hosting. It doesn’t work properly as there is no PHP mail function and there is no ability to use formated permalinks. Seems like a poor business choice to offer something that’s basic functionality isn’t supported.</p>
Integrated calendaring and the quest for simple complexity2005-06-08T00:00:00+00:00http://joshuami.com/2005/06/08/integrated-calendaring-quest-simple-complexity<p>I would imagine the human factor associated with adopting a centralized calendar—or for that matter a content management system—becomes more complex the larger the institution gets. On the other hand, the larger institution should have more resources and personnel to throw at the issue–though this is not always the case.<!--more--></p>
<p>As Web Manager for <a href="http://www.pacificu.edu/">Pacific University</a> (approximately 2,500 students), my solution for calendaring is an internally-built application that I call CANS (<a href="http://www.pacificu.edu/cansinfo/">Calendar And News System</a>). As the name implies, it also handles news/articles. It is basically the second time that I’ve written this app. The first time was for <a href="http://www.cu-portland.edu/">Concordia University - Portland, Oregon</a>.</p>
<p>The first CANS was written with Coldfusion and MS SQL on a couple of Microsoft servers; the second with Coldfusion and MySQL in a Linux environment.</p>
<p>The backend is unimportant—I could have written it in [insert coding language here] and [insert flavor of database here] with similar functionality–the core requirement for implementing CANS is a relational database. (Sorry folks, you can’t get away with using an old version of FileMaker.)</p>
<p>What makes the conceptual model worthy of considering is the way that it treats “departments” (functional units of the institution) in relation to “categories” (roughly divided into unique audience groups that would be associated with a specific online resource such as a webpage or email newsletter).</p>
<p>Institutional units are usually quick to latch on to CANS because they see it as a benefit to their unit’s website. Some departments have several child categories associated with a parent category allowing them to get specific with the information they want to present.</p>
<p>This doesn’t make much sense without an example. Take a look at the <a href="http://www.pacificu.edu/career/">career center at Pacific</a>. The “Career Development Center” is a top level (parent) category with subcategories (children) of “Job Fairs” and “Spotlight on Success”. The output code calls up the events (a row in the CALENDAR table) or articles (a row in the NEWS table) for a specific category (a row in the CATEGORY table). Linking tables hold things together (i.e. NEWS2CATEGORY). This makes it possible to link multiple categories to a single event so that it will show up on several pages of the website as appropriate.</p>
<p>For the staff/faculty/students that log into the system (integrated LDAP) and add events, they are presented with the master calendar where they must look at the day associated with their event before they can click the “add” button. (This helps reduce redundancy and scheduling conflicts.) They will then see a simple form with some required fields including a multiple select box that asks “Where do you want this to appear?”.</p>
<p>For website visitor, there is no real need to know that any of this is happening. They simply see news and events on that pertain to topic covered by the webpage on which they landed.</p>
<p>The model is inherently expandable. The Concordia version has image upload capabilities and a special interface for athletics to manage their schedule and results with additional fields such as opponent, outcome and final score.</p>
<p>As an application, CANS is far from perfect. (Any software written by a single person—with no less than a degree in English—who is grossly overburdened with other design and development projects can’t help but have a few bugs.) However, it has enabled me to get people across campus, both here and at Concordia, to focus on communicating to their audience(s) with timely and useful information.</p>
<p>Incidentally, CANS does not currently integrate with the Pacific’s scheduling software—though I do plan to explore this someday. I would love to have the functionality of a tool such as <a href="http://corp.collegenet.com/depts/higher_ed/series/R25/">R25</a>, but it would add another layer of complexity that would be too much for me to take on at this time.</p>
<p>I would also, if I had the time and personnel, like to integrate the application with a full content management system. Anyone who wants to work with me on the model is free to drop me a line.</p>
Which OS Are You?2005-05-20T00:00:00+00:00http://joshuami.com/2005/05/20/which-os-are-you<p>Woohoo! I’m not Windows Me!</p>
<p><a href="http://bbspot.com/News/2003/01/os_quiz.php"><img src="http://www.bbspot.com/Images/News_Features/2003/01/os_quiz/redhat.jpg" alt="You are Red Hat Linux. You're tops among your peers, but still get no respect from them. It's all right with you. You have your sights set higher." />
Which OS are You?</a></p>
<p><a href="http://bbspot.com/News/2003/01/os_quiz_all.html">See the other possibilities</a>.</p>
Which File Extension Are You?2005-05-20T00:00:00+00:00http://joshuami.com/2005/05/20/which-file-extension-are-you<p>I am so glad it is Friday.</p>
<p><a href="http://www.bbspot.com/News/2004/10/extension_quiz.php"><img src="http://www.bbspot.com/Images/News_Features/2004/10/file_extensions/star.jpg" /></a></p>
<p>You are .* You are a wildcard. You are everything to everybody. You can’t make up your mind as to what you want to be.</p>
<p><a href=""http://bbspot.com/News/2004/10/extension_quiz_all.html"">Possible results</a>.</p>
I'm a homeowner2005-04-29T00:00:00+00:00http://joshuami.com/2005/04/29/homeowner<p>Well, it might be more appropriate to say that we, my wife and I, are considerably indebted to the bank with a small amount of equity due to an excellent purchase price.</p>
<p>The hurry-up-and-wait process ended this morning in a flurry of activity that began with a phone call late yesterday afternoon.</p>
<p>So to sum up the week, my daughter couldn't hold down anything on Thursday; the title company called yesterday and wanted us to sign with two hours notice—we declined until this morning due to "scheduling" issues, but were fine with signing first thing this morning; which we did, despite the fact I was awoken this morning by my son's projectile vomitting in my face. (Yes, you read that correctly.)</p>
<p>The next two weeks are guaranteed to be crazy, but at least I have my Internet connection back—oh yes, that is another story for another day.</p>
90% of college freshmen have never shared a room2005-04-27T00:00:00+00:00http://joshuami.com/2005/04/27/90-percent-college-freshman-never-shared-room<p>When I heard this statistic—which is a statistic for Pacific University, though likely to be fairly typical—I couldn't help but be a little discouraged by it. Pacific's growth plans include the development of residence halls (dorms to anyone not used to higher-education speak) with rooms that will house individuals—no roommates. If 90% of incoming students have never had to share space and resources with a sibling, how can we expect them to share space and resources on a larger scale.</p>
<p>While I will readily admit to having a room of my own for the majority of my middle and high school years, I experienced the benefit of compromise-laden living with my younger brother for about 10 years of our lives together. Including my married years, I can say that I have spent twice as many years sharing a bedroom as not. There is a lot to be gleaned from the ability to share a relatively small space on a regular basis. With siblings, or with a life partner, there will inevitably be arguments and clashes of opinion, but it is the response to these conflicts that helps to shape our emotional intelligence.</p>
<p>While I believe you can learn empathy in a number of ways, it seems to be that the lessons learned from compromising with a roommate could be extremely valuable. While there may be some merit to the solitude that an individual room in a residence hall could afford, I would hope that the incoming freshmen would be encouraged to share a little of themselves in a way more intimate than instant messaging—which increasingly seems to be the choice of the college-age crowd.</p>
Daring Fireball - Translation From PR-Speak to English of Selected Portions of Adobe's 'FAQ' Regarding Their Acquisition of Macromedia2005-04-22T00:00:00+00:00http://joshuami.com/2005/04/22/translation-from-pr-speak<p><a href="http://daringfireball.net/2005/04/adobe_translation">Daring Fireball: Translation From PR-Speak to English of Selected Portions of Adobe’s ‘FAQ’ Regarding Their Acquisition of Macromedia</a> - Made me laugh. I wish all legalese could be translated with such acumen. Did I mention I’m buying a house?</p>
House hunting and waiting2005-04-05T00:00:00+00:00http://joshuami.com/2005/04/05/house-hunting-and-waiting<p>This weekend my wife and I went house hunting. We thought we had our loan in hand—more on that latter—so it should have been as easy as finding something we liked and going from there. It wasn't quite so easy.</p>
<p>First, there is the little issue of how much home we could afford. We didn't want to stretch our budget too far. Unfortunately, an entire Saturday was spent depressing ourselves with just how expensive property values in Portland are becoming—or more appropriately how run down and off the beaten path an affordable home is becoming. We finished the day with a short list that wasn't all that reassuring.</p>
<p>With thoughts of rent dancing in our heads, we visited the first home on our list. Ack! While my kids thought an empty house with oddly shaped rooms and a floor that echos when you jump was great, Diana and I are a little more discerning. The house was at the top of our range, and we thought, "well, with another $20,000 we could make it livable." One down.</p>
<p>The second place was a long shot. We were only looking at it because it had three bedrooms. The outside was/is...1950's California ranch. Not bad from the street, but not the height of architectural elegance either. The drive by the day before was a fading memory, and we kept getting it confused with the 15 other houses we'd looked at. Then we walked up to it. Nice flower bed on the front of the house, well trimmed yard, it was beginning to grow on us. When we stepped inside, our attitude towards the day did a 180.</p>
<p>Suddenly, the thought of home ownership was exciting. The enterior had a few updates, but more importantly it was clean and spacious and had a kitchen that met all our requirements. The backyard was huge by urban standards. The only thing it lacks is a garage for the bikes, but it has storage and we could easily replace the carport with a garage.</p>
<p>It all seemed too good to be true. We followed the realtor to one more house that we didn't even want to see, then followed her out to her office to fill out the paperwork to make an offer. Things seemed to go from a lull to full speed in a matter of hours.</p>
<p>The next day, we were a ball of nerves. The realtor called to say the offer was in and the current owners were talking to their realtor. What seemed like days later, but was really just later that evening—when do they take off—the realtor called to say the offer was accepted. We were in the neighborhood—despite the new neighborhood being a 20 minute drive from our current townhome—and were thrilled.</p>
<p>Now, I'm sitting at the computer a little over 24 hours later telling this story. We still don't know if the whole thing will work out. The lender hasn't got back to us with some needed paperwork that we have to have by 2:30 p.m. tomorrow. We are stressed and excited and nervous and everything else you can be.</p>
Weblog of the Mobile Hippie2005-03-28T00:00:00+00:00http://joshuami.com/2005/03/28/weblog-of-mobilehippie<p>The <a href=""http://www.mobilehippie.com/"">Weblog of the Mobile Hippie</a> has just received a much needed visual update.</p>
<p>I had been wanting to add comments to my blog for sometime, but due to the time output—however little that may have been—I was slow to get around to it. Well, now I have comments, a new navigation bar, and a slick flexible layout that takes advantage of <code>max-width</code> to limit line length and hold the layout together—Internet Explorer can't see this.</p>
<p>While I know there is a <a href=""http://www.svendtofte.com/code/max_width_in_ie/"">hack to simulate max-width in IE</a> using <code>expression()</code>, but I'm hesitant to use it as it crashed the browser several times during testing. This glitch may be due to my installation of multiple instances of IE. Multiple installations has created other glitches for me in the past—including a shifting text bug that nearly caused me to throw out a whole CSS file until I realized that it could not be duplicated on my home computer or coworkers computers. The only difference in setup on my home computer was the multiple installs of IE. So the moral of this story is that the only thing worse than one install of IE is multiple installs of IE. Unfortunately, I can't give up the multi-install as I don't want to run a separate OS for IE 5.01 and IE 5.5, and with their significant share of the market, they must be tested for.</p>
<p>Incidentally, you are out of luck if you are using an older version of IE on this site. I haven't included a high-pass filter and CSS overrides for these browsers—my personal site is a much, much lower priority than the sites that make up my day job.</p>