Update 11/11/2024: The hugo deploy command was introduced which allows you to handle this entirely using Hugo. The pipeline below could probably still work but I’m no longer using it.

Previously, this post described how to create a working Bitbucket pipeline for deploying a Hugo static site to AWS S3.

I recently decided to migrate my blog from Wordpress to Hugo. I don’t blog as often these days so trying to keep up with new wordpress versions, securing my wordpress installation against hack attempts, and fighting comment spam was becoming overwhelming.

My latest project in AWS has involved AWS API Gateway and AWS Lambda deployed using Serverless Framework. The integration between AWS Lambda and API Gateway is a little tricky to get exactly right, so I’m documenting some key findings here.

I was working a Scala big data project with a lot of dependencies. Most of the time the dependencies exist on the server already or get packaged up inside an uber jar. In this case, the server was missing the dependencies and I didn’t want to package everything in the uber jar due to super slow upload speeds. This combined with the need to do a bunch of trial and error meant I wanted to upload all the dependencies to the server and then just tweak my few lines of code. I ended up using a nice maven command combined with a few lines of python to gather all of them up in one place.

When working in a corporate environment, you’ll often have to deal with self-signed certificates that are used to secure internal dev tools like Artifactory or a git server.

A collection of code snippets and notes from working with linux.

I needed to be able to the run Spark’s shell using a different home directory due to a permissions issue with my normal HDFS home directory. A little known feature with spark is that it allows overriding Hadoop configuration properties!

Running Apache Storm in production requires increasing the nofile and nproc default ulimits.

I recently ran into an issue submitting Spark applications to a HDInsight cluster. The job would run fine until it attempted to use files in blob storage and then blow up with an exception: java.lang.ClassCastException: org.apache.xerces.parsers.XIncludeAwareParserConfiguration cannot be cast to org.apache.xerces.xni.parser.XMLParserConfiguration.

My goal was to create a process for importing data into Hive using Sqoop 1.4.6. It needs to be simple (or easily automated) and use a robust file format.

Importing data from a Relational Database into Hive should be easy. It’s not. When you’re first getting started there are a lot of snags along the way including configuration issues, missing jar files, formatting problems, schema issues, data type conversion, and … the list goes on. This post shines some light on a way to use command line tools to import data as Avro files and create Hive schemas plus solutions for some of the problems along the way.

Spark includes the ability to write multiple different file formats to HDFS. One of those is ORC which is columnar file format featuring great compression and improved query performance through Hive.

There is a lot of documentation around Apache Storm and Apache HBase but not so much about how to use the hbase-client inside of storm. In this post, I’ll outline:

1. Information about my dev environment
2. Setting up your Storm project to use the HBase client
3. Managing connections to HBase in Storm
4. Reading one row (Get)
5. Reading many rows (Scan)
6. Writing one row (Put)
7. Writing many rows in a batch of Puts

Please note, this post assumes you already are comfortable with Storm and HBase terminology. If you are just starting out with Storm, take a look at my example project on GitHub: storm-stlhug-demo.

Also, an option to consider when writing to HBase from storm is storm-hbase and it is a great way to start streaming data into hbase. However, if you need to write to multiple tables or get into more advanced scenarios you will need to understand how to write your own HBase bolts.

I ran into some trouble executing a simple MapReduce program on TEZ. I kept reading about the special “-Dmapreduce.framework.name=yarn-tez” parameter you could pass to your MR job to make it automatically switch frameworks without modifying the configs for your entire cluster but couldn’t get the argument to be parsed correctly.

After doing some research, I found that your MapReduce class must extend Configured and implement Tool. In addition to parsing the generic arguments correctly, you also get arguments parsed for you automatically.

A few notes from playing with zeppelin on the Hortonworks HDP 2.3 sandbox.

Logging within storm uses Simple Logging Facade for Java (SLF4J).

Tick tuples are useful when you want to execute some logic within your bolts on a schedule. For example, refreshing a cache every 5 minutes.

Within your bolt, first enable receiving the tick tuples and configure how often you want to receive them:

@Override
public Map<String, Object> getComponentConfiguration() {
    // configure how often a tick tuple will be sent to our bolt
    Config conf = new Config();
    conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 300);
    return conf;
}

Next create the isTickTuple helper method to determine whether or not we’ve received a tick tuple:

As we gradually replace regular windows command line with powershell, it will be useful to set up a powershell environment for Java / Maven development.

We leverage Reporting Services subscriptions a lot to deliver reports via email. A problem we run into frequently is when the user who created the subscription loses permission (leaving the company, changing jobs, etc.) then all of the subscriptions they created stop working. This is because they are the Owner of the subscription and SSRS won’t allow the subscription to run when the owner no longer has permission.

Here is a quick powershell script you can use to change the owner using the ReportingService2010 web service endpoint.

First, make sure you have subscribed to the Microsoft Translator API on Azure Marketplace and registered your application Azure DataMarket. For testing, you can use the free subscription which lets you submit up to 2 million characters a month. Registering your app will create a Client ID and a Client Secret which you need to get authorization.

In order to protect your Client Secret, you’ll need to write server-side code to get your access token. The access code is then passed to the Microsoft Translator API via AJAX.

I wrote a small python script to help convert Handy Safe to Sky Wallet. I definitely learned a lot about python and even though I wasn’t very familiar with the available API I was able to come up with a pretty nifty script.

I started helping my brother with some GO projects. One really interesting concept in Go are slices… which make working with arrays easier.

At anytime you can “chop” out interesting portions of a slice to use using [start:finish].

There is the special append keyword to add things onto the end of the slice dynamically (as opposed to using the make keyword to declare it up front).

And there is a range keyword to mimic how foreach works in a higher level language.

I had an issue trying to add some minified JavaScript files to ClearCase.

It looks like files that have over 8000 characters on a single row are unable to be added to version control using normal methods.

Thankfully, I was able to find a solution here:

Adding the file to clearcase as a “compressed file” means it doesn’t try to do a file delta!

A list of gripes about features I feel are missing from Microsoft SQL Server Reporting Services 2008 R2.

After my botched Windows 8 install, I decided to go back to Ubuntu. I still would like to figure out a solution with Windows on this laptop so that I could use it for work but right now having linux is better. I decided to document some of the issues I ran into in case I need them later.

I ran across a strange error while deploying a very simple DLL. It contained only one class; a web services proxy class to the Lists SharePoint web service that I generated using the wsdl tool. Here is the error:

Unable to generate a temporary class (result=1).

error CS2001: Source file ‘C:\WINDOWS\TEMP_zm3ve2y.0.cs’ could not be found

error CS2008: No inputs specified

I want to put a quick post out here in support of SharePoint Overflow being migrated to StackExchange v2.0. It is a a great complement to StackOverflow for all things SharePoint related.

You can commit here: SharePoint Overflow.

One thing that bugs me all the time is when a window doesn’t remember the position that it was resized or moved to. Although it is more of a nuisance than anything else, I decided to write a little AutoIt3 script to automatically resize my Remote Desktop windows to be the certain place and size that I prefer them to be in.

For SQL Server Reporting Services (SSRS) reports there is the ability to create subscriptions. Subscriptions can be scheduled to run on a certain schedule to send emails, export reports to SharePoint document libraries, or save to windows file shares.

Depending on your requirement, you may need to grant or remove access to the subscribe action and can be managed by creating or editing the default SharePoint roles: Read, Contribute, Full Control, etc.

A list of SharePoint 2013 LCIDs along with their language and the time format.

InfoPath is a common tool used to create forms hosted on SharePoint. Due to the limited development options available for Office 365, I think this places an increased focus on InfoPath when it comes to creating a solution. Unfortunately, I found that it was relatively difficult to get the current user’s information to be displayed or saved within an Infopath form.

I am working on a small query tool that will be used to make AJAX calls against SharePoint’s Enterprise Search Query Web Service. Previously, I created the SOAP messages by appending a gigantic series of strings together. In this iteration, I wanted to use JsRender (my template engine of choice right now and the successor to jQuery templates).

Imagine you work for a large company that uses Microsoft Project Server. Since you have many employees, having MSP sync individual permissions for every user to every site is simply not scalable. Instead, you’d like to use some Active Directory groups that are already set up to manage permissions in the workspace.

So you disable the workspace permissions sync in PWA:

1. Navigate to PWA
2. Click Server Settings
3. Click Project Workspace Provisioning Settings
4. Validate that the “Workspace Permissions” checkbox is unchecked

Then, perhaps you write some code that is wrapped up in a nice SharePoint feature to manage setting up the permissions yourself. Everything works great….

… until you start getting reports that some user’s permissions aren’t quite right for some project sites.

I was writing a little web services app to help delete individual permissions from a site and thought about trying XPATH on the result to get what I needed. I had some trouble so thought I’d put together a small example for later reference.

Quick update to SPUtility.js today! I’ve added support for rich text fields, unchecking multi-select checkboxes, and choice fields with fill-in values. I also have some thoughts on future SPUtility.js updates.

Version 0.5 of SPUtility.js has been released onto the codeplex site! Go check it out!

After recently doing a lot of work heavily customizing NewForm.aspx and EditForm.aspx for a SharePoint list, I ran into a major issue after I moved my list into a publishing site. The issue appeared when I added a webpart onto my EditForm.

SPUtility.js version 0.4 should be getting released soon (week or so??). I wanted to put a quick post out there to highlight some of the upcoming features and fixes.

SPUtility.js is a library used to get/set SharePoint’s form fields. We can also use it autopopulate a Newform/Editform based on some URL parameters which are passed to the page!

I recently got asked if it was possible wrap the column headers in a list view web part. The user had a WebPart with quite a few columns with long names and was trying to prevent the page from scrolling left to right.

In this era of Web 2.0 web applications, I think it was only a matter of time before I needed to investigate how I could apply AJAX to SharePoint. With a little JavaScript and SharePoint’s web services, you can build fancy apps that can easily be dropped into a document library. And after reading Glen Cooper’s blog post on the subject, I figured it was something definitely worth investigating.

I also took this opportunity to improve a process I have to do all the time: what are the internal column names in a list? To get the list information, I built a simple AJAX app that consumes the Lists.asmx web service. Please keep in mind that this code will have be uploaded into SharePoint in order to work (a document library will work) and will not run from your desktop.

November 10, 2010 update: I’ve posted the sequel to this post: Autopopulate a SharePoint Form from URL (with SPUtility.js). I highly recommend checking out the new and improved way using SPUtility.js!

This post builds off of my previous post about making fields read only. In this case, I needed to autopopulate the NewForm.aspx with a value that gets passed to the form via URL parameters.

The goal of this post is to provide an easy to understand example of how to enable editing in an SPGridView. In this case, my datasource of choice is a SharePoint list.

The way that I chose to implement editing in an SPGridView involved two classes:

• SimpleLogic – handles query and update operations
• SimpleSPGrid – handles all of the setup for the ObjectDataSource and the SPGridView

The main advantage for splitting the two classes is to separate all of the business logic into one class. Ideally, you should be able to completely change how the SimpleLogic class retrieves/updates data without changing the SimpleSPGrid code.

If you’ve done any work with SharePoint and SPAuditEntry, you may have noticed the distinct lack of Create event. Instead, two Update events are created which means some extra processing is required.

Add this to the list of things every SharePoint developer should know (up there with disposing SPWebs and SPSites).

In general…

1. Don’t update SharePoint objects on a GET request
2. Call SPUtility.ValidateFormDigest() before anything on a POST request

Here are the two links to read:

• What you need to know about AllowUnsafeUpdates (Part 1)
• What you need to know about AllowUnsafeUpdates (Part 2)

Two identical custom lists with the same columns. The only difference is that one list has columns added using Site Columns and the other list had columns added directly to the list. The same? Not so much…

**May 20, 2010 Update

** This post is about using a C# console app to update choice fields via SharePoint web services.  If you are looking for more information on how SharePoint web services can be called via JavaScript, check out my post on building a simple SharePoint AJAX app.

My problem started with a simple change to a site column. I had originally created a Country field with the following values:

• US
• Canada

After a change in requirements, the new values for the field were:

• Canada
• Germany
• Great Britain
• Ireland
• United States

After updating the site column, I then needed to update the hundreds of items in the list to use “United States” instead of “US”. Instead of updating hundreds of items manually, I decided to write a small web services console app. (I was also not able to update it in datasheet… due to content approval maybe???)

However, I quickly ran into problems when attempting to update the multi-select choice field.

May 20, 2010 Update

If you are looking for a great example on how to create an SPGridView, check out Erik Burger’s blog:

1. Building A SPGridView Control – Part 1: Introducing the SPGridView
2. Building A SPGridView Control – Part 2: Filtering

This post is really meant to supplement part 2 of Erik’s guide and address values in the grid that include apostrophes when using filtering.

The SPGridView is one of the most useful SharePoint controls. You can use it to do sorting, grouping, and filtering just like the out of the box List View Web Part does for regular lists and libraries. This makes it relatively easy to create a custom grid with your own data. Unfortunately, filtering with the SPGridView is a little quirky, especially if your data could potentially contain apostrophes.

The most recent problem I ran into was when I tried to filter out items that had a  certain Workflow status in the list. More specifically, I was trying to filter on items that had errored out and had a status of “Error Occurred” in the list.

I recently ran into a problem creating alerts in a web part for the current user. After the alert was added, no matter what I set the SPAlert’s EventType to, it always defaulted to All. I finally figured out a way around this problem thanks to a helpful post by erwin at sharepointology.

While researching a bug with editing in datasheet view and a custom list event handler, I discovered strange behavior with boolean values. Any boolean value you get from the SPItemEventDataCollection depends on where the item was originally edited: either using the regular user interface (EditForm.aspx, NewForm.aspx) or using datasheet view.

When creating a workflow using SharePoint Designer, oftentimes it is useful to be able to create a link to the current list item. Unfortunately, the built in columns are not so helpful.

I recently ran into the weird issue of not having my tabs highlighted correctly. After consulting Uncle Google, it seems Andy Burns ran into this same issue.

The problem started after creating multiple aspx pages within my site and then setting up a tab for each page by editing the Top Link Bar (also known as the top navigation) under Site Settings. Each of the tabs worked correctly, but after clicking a tab that tab would not be highlighted and the home tab would stay selected.

Recently I ran into a “feature” of SharePoint’s SPSite.OpenWeb() method (the no argument constructor specifically). If the OpenWeb() method is used with a URL that is not known to exist, it can result in some unexpected behavior.

November 10, 2010 Update: I’ve posted the sequel to this post: Autopopulate a SharePoint Form from URL (with SPUtility.js). The code below will not work in SharePoint 2013 so you should use SPUtility.js!

After looking at a couple of examples online, most of them require the use of SharePoint Designer to make a field read only (or to hide it). Here is a method to make a field readonly using only Prototype (javascript library) and a Content Editor Web Part

Got the following message when trying to deploy a solution file to the server:

Failed to extract the cab file in the solution.

Luckily Robert Bogue had already found the answer: we had duplicated an entry for a web part in our solution’s .ddf file. Apparently, makecab allows you to have the same file in the same location multiple times. I’m not really sure why it doesn’t just overwrite it, but in any case, the solution doesn’t know what to do with it either.

I’ve learned a couple of different things from messing with Enterprise Search and SharePoint. A lot of this has to do with the initial setup and how different crawls affect the index.

What this assumes:

• You know how to get to Central Admin and the Search Admin screens

• Any URLs below you will have to replace “centraladmin” with your own server ip

• The place where you can run incremental or full crawls is:

Tinkers’ Construct is a mod for Minecraft.

Tinkers’ Construct Materials

The following materials are what you use to build your tools.

• Reinforced: 10% chance per level of not using durability.
• Stonebound: The tool mines faster as it wears out, but does less damage.
• Jagged: The tool does more damage as it wears out, but mines slower.
• Writable: One extra modifier per piece.
• Tasty: Sometimes drops food items.
• Slimy: Randomly spawns small slimes.

Tinkers’ Construct Alloys

By combining different metals inside the Smeltery, you can create alloys.

A simple rule for validating an observable which could contain multiple email addresses.

Assuming you have an observable like self.emailTO = ko.observable('');, you would just extend it to have self.emailTO.extend({ multiemail: true });. Use it with the required validator if you want at least one email address.

ko.validation.rules['multiemail'] = {
    validator: function (val, validate) {
        if (!validate) { return true; }

        var isValid = true;
        if (!ko.validation.utils.isEmptyVal(val)) {
            // use the required: true property if you don't want to accept empty values
            var values = val.split(';');
            $(values).each(function (index) {
                isValid = ko.validation.rules['email'].validator($.trim(this), validate);
                return isValid; // short circuit each loop if invalid
            });
        }
        return isValid;
    },
    message: 'Please enter valid email addresses (separate multiple email addresses using a semicolon).'
};

Requires Knockout.js, Knockout Validation, and jQuery.

I am working on a small query tool that will be used to make AJAX calls against SharePoint’s Enterprise Search Query Web Service. Previously, I created the SOAP messages by appending a gigantic series of strings together. In this iteration, I wanted to use JsRender (my template engine of choice right now and the successor to jQuery templates).

I’ve been learning a lot about JsRender and Knockout.js after finding Ryan’s answer on StackOverflow. I’ve created a couple of templates and started to use the {{for}}{{/for}} tag.

I was having trouble figuring out how to access the parent object’s data from within the for loop. After reading John Papa’s awesome posts, Using JsRender with JavaScript and HTML and Advanced JsRender Templating Features I came up with a quick custom “Debug” tag to help me understand.

My team has recently created quite a few awesome WCF services that can surface data via REST or JSON. The next step, was building an interface on top of the services using AJAX.

Since the services were hosted on a SharePoint farm, it is super easy to just upload an HTML file into a document library and start coding away. However, eventually this process breaks down; I needed a way to deploy these files to different environments and keep them in some sort of version control system. (No, a document library doesn’t count as version control)

Working with JavaScript can definitely be painful, but I always love stumbling across interesting and strange features of the language. I just found wtfjs.com which led me to these two answers on stackoverflow.

I was developing an input form for a search when I came across the need to allow the user to filter items using a date range. I had two input boxes where the user could enter a start and an end date. For some of the most common date ranges, I decided to create links that could be clicked to auto-populate the start/end date textboxes. For example, searching for items that were created last week or last month. Using JavaScript’s Date object, I was able to put together some nice examples.

Strangely, Prototype’s documentation for Element.insert does not have any examples, so I cooked something up quick:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
        "http://www.w3.org/TR/html4/loose.dtd">

<html>

<head>

<title>Template</title>

<script type="text/javascript" src="prototype.js"></script>
<script type="text/javascript">

Event.observe(window, 'load', function() {
    // Using Element.insert to insert after, before, top, or bottom
    Element.insert($('para1'), { before: "<p>This is paragraph zero.</p>" });
    Element.insert($('para1'), { after: "<p>This is paragraph two.</p>" });
   
    // notice we have to insert using the content container
    Element.insert($('content'), { top: "<p>This is the first paragraph.</p>" });
    Element.insert($('content'), { bottom: "<p>This is the last paragraph</p>" });
});

</script>

</head>

<body>

<div id="content">
    <p id="para1">This is paragraph one.</p>
    <p id="para3">This is paragraph three.</p>
</div>

</body>
</html>