Category: Cloud
Integrating AWS Lambda with API Gateway in JavaScript
My latest project in AWS has involved AWS API Gateway and AWS Lambda deployed using Serverless Framework. The integration between AWS Lambda and API Gateway is a little tricky to get exactly right, so I’m documenting some key findings here.
Category: Hadoop
Run Spark Shell with a Different User Directory
I needed to be able to the run Spark’s shell using a different home directory due to a permissions issue with my normal HDFS home directory. A little known feature with spark is that it allows overriding Hadoop configuration properties!
Apache Storm and ulimits
Running Apache Storm in production requires increasing the nofile and nproc default ulimits.
ClassCastException submitting Spark apps to HDInsight
I recently ran into an issue submitting Spark applications to a HDInsight cluster. The job would run fine until it attempted to use files in blob storage and then blow up with an exception: java.lang.ClassCastException: org.apache.xerces.parsers.XIncludeAwareParserConfiguration cannot be cast to org.apache.xerces.xni.parser.XMLParserConfiguration
.
Creating an Avro table in Hive automatically
My goal was to create a process for importing data into Hive using Sqoop 1.4.6. It needs to be simple (or easily automated) and use a robust file format.
Importing data from a Relational Database into Hive should be easy. It’s not. When you’re first getting started there are a lot of snags along the way including configuration issues, missing jar files, formatting problems, schema issues, data type conversion, and … the list goes on. This post shines some light on a way to use command line tools to import data as Avro files and create Hive schemas plus solutions for some of the problems along the way.
Writing a Spark DataFrame to ORC files
Spark includes the ability to write multiple different file formats to HDFS. One of those is ORC which is columnar file format featuring great compression and improved query performance through Hive.
Using HBase within Storm
There is a lot of documentation around Apache Storm and Apache HBase but not so much about how to use the hbase-client inside of storm. In this post, I’ll outline:
- Information about my dev environment
- Setting up your Storm project to use the HBase client
- Managing connections to HBase in Storm
- Reading one row (Get)
- Reading many rows (Scan)
- Writing one row (Put)
- Writing many rows in a batch of Puts
Please note, this post assumes you already are comfortable with Storm and HBase terminology. If you are just starting out with Storm, take a look at my example project on GitHub: storm-stlhug-demo.
Also, an option to consider when writing to HBase from storm is storm-hbase and it is a great way to start streaming data into hbase. However, if you need to write to multiple tables or get into more advanced scenarios you will need to understand how to write your own HBase bolts.
Example running MapReduce on tez and parsing arguments
I ran into some trouble executing a simple MapReduce program on TEZ. I kept reading about the special “-Dmapreduce.framework.name=yarn-tez” parameter you could pass to your MR job to make it automatically switch frameworks without modifying the configs for your entire cluster but couldn’t get the argument to be parsed correctly.
After doing some research, I found that your MapReduce class must extend Configured and implement Tool. In addition to parsing the generic arguments correctly, you also get arguments parsed for you automatically.
Apache Zeppelin on the Hortonworks 2.3 sandbox
A few notes from playing with zeppelin on the Hortonworks HDP 2.3 sandbox.
Getting started with Storm: Logging
Logging within storm uses Simple Logging Facade for Java (SLF4J).
Tick tuples within Storm
Category: JavaScript
Knockout Validation: Multiple email addresses
Using JsRender to create SOAP Envelopes
I am working on a small query tool that will be used to make AJAX calls against SharePoint’s Enterprise Search Query Web Service. Previously, I created the SOAP messages by appending a gigantic series of strings together. In this iteration, I wanted to use JsRender (my template engine of choice right now and the successor to jQuery templates).
Custom Debug JsRender tag
I’ve been learning a lot about JsRender and Knockout.js after finding Ryan’s answer on StackOverflow. I’ve created a couple of templates and started to use the {{for}}{{/for}}
tag.
I was having trouble figuring out how to access the parent object’s data from within the for loop. After reading John Papa’s awesome posts, Using JsRender with JavaScript and HTML and Advanced JsRender Templating Features I came up with a quick custom “Debug” tag to help me understand.
AJAX app development using Visual Studio and a remote WCF service
My team has recently created quite a few awesome WCF services that can surface data via REST or JSON. The next step, was building an interface on top of the services using AJAX.
Since the services were hosted on a SharePoint farm, it is super easy to just upload an HTML file into a document library and start coding away. However, eventually this process breaks down; I needed a way to deploy these files to different environments and keep them in some sort of version control system. (No, a document library doesn’t count as version control)
Relative Dates in JavaScript
I was developing an input form for a search when I came across the need to allow the user to filter items using a date range. I had two input boxes where the user could enter a start and an end date. For some of the most common date ranges, I decided to create links that could be clicked to auto-populate the start/end date textboxes. For example, searching for items that were created last week or last month. Using JavaScript’s Date object, I was able to put together some nice examples.
Using Prototype's Element.insert
Category: Minecraft
Tinkers' Construct Cheat Sheet
Category: Notes
How to list every Maven Dependency for a Project
I was working a Scala big data project with a lot of dependencies. Most of the time the dependencies exist on the server already or get packaged up inside an uber jar. In this case, the server was missing the dependencies and I didn’t want to package everything in the uber jar due to super slow upload speeds. This combined with the need to do a bunch of trial and error meant I wanted to upload all the dependencies to the server and then just tweak my few lines of code. I ended up using a nice maven command combined with a few lines of python to gather all of them up in one place.
Dealing with Self-signed Certificates
When working in a corporate environment, you’ll often have to deal with self-signed certificates that are used to secure internal dev tools like Artifactory or a git server.
Category: SharePoint
SSRS Subscribe Action and SharePoint Permissions
For SQL Server Reporting Services (SSRS) reports there is the ability to create subscriptions. Subscriptions can be scheduled to run on a certain schedule to send emails, export reports to SharePoint document libraries, or save to windows file shares.
Depending on your requirement, you may need to grant or remove access to the subscribe action and can be managed by creating or editing the default SharePoint roles: Read, Contribute, Full Control, etc.
SharePoint 2013 LCIDs with default Time Format
A list of SharePoint 2013 LCIDs along with their language and the time format.
Pass user information from SharePoint 2013 to InfoPath 2013
InfoPath is a common tool used to create forms hosted on SharePoint. Due to the limited development options available for Office 365, I think this places an increased focus on InfoPath when it comes to creating a solution. Unfortunately, I found that it was relatively difficult to get the current user’s information to be displayed or saved within an Infopath form.
Using JsRender to create SOAP Envelopes
I am working on a small query tool that will be used to make AJAX calls against SharePoint’s Enterprise Search Query Web Service. Previously, I created the SOAP messages by appending a gigantic series of strings together. In this iteration, I wanted to use JsRender (my template engine of choice right now and the successor to jQuery templates).
Workspace permissions and Project.QueuePublish
Imagine you work for a large company that uses Microsoft Project Server. Since you have many employees, having MSP sync individual permissions for every user to every site is simply not scalable. Instead, you’d like to use some Active Directory groups that are already set up to manage permissions in the workspace.
So you disable the workspace permissions sync in PWA:
- Navigate to PWA
- Click Server Settings
- Click Project Workspace Provisioning Settings
- Validate that the “Workspace Permissions” checkbox is unchecked
Then, perhaps you write some code that is wrapped up in a nice SharePoint feature to manage setting up the permissions yourself. Everything works great….
… until you start getting reports that some user’s permissions aren’t quite right for some project sites.
Web services, XmlNode, and XPATH
I was writing a little web services app to help delete individual permissions from a site and thought about trying XPATH on the result to get what I needed. I had some trouble so thought I’d put together a small example for later reference.
SPUtility.js 0.8 and beyond!
Quick update to SPUtility.js today! I’ve added support for rich text fields, unchecking multi-select checkboxes, and choice fields with fill-in values. I also have some thoughts on future SPUtility.js updates.
SPUtility 0.5 Released!
Version 0.5 of SPUtility.js has been released onto the codeplex site! Go check it out!
Customizing EditForm.aspx in a Publishing site
After recently doing a lot of work heavily customizing NewForm.aspx and EditForm.aspx for a SharePoint list, I ran into a major issue after I moved my list into a publishing site. The issue appeared when I added a webpart onto my EditForm.
Planned updates for SPUtility.js 0.4
SPUtility.js version 0.4 should be getting released soon (week or so??). I wanted to put a quick post out there to highlight some of the upcoming features and fixes.
Autopopulate a SharePoint Form from URL (with SPUtility.js)
SPUtility.js is a library used to get/set SharePoint’s form fields. We can also use it autopopulate a Newform/Editform based on some URL parameters which are passed to the page!
Wrap ListViewWebPart Column Headers
I recently got asked if it was possible wrap the column headers in a list view web part. The user had a WebPart with quite a few columns with long names and was trying to prevent the page from scrolling left to right.
A Simple AJAX App using SharePoint Web Services
In this era of Web 2.0 web applications, I think it was only a matter of time before I needed to investigate how I could apply AJAX to SharePoint. With a little JavaScript and SharePoint’s web services, you can build fancy apps that can easily be dropped into a document library. And after reading Glen Cooper’s blog post on the subject, I figured it was something definitely worth investigating.
I also took this opportunity to improve a process I have to do all the time: what are the internal column names in a list? To get the list information, I built a simple AJAX app that consumes the Lists.asmx web service. Please keep in mind that this code will have be uploaded into SharePoint in order to work (a document library will work) and will not run from your desktop.
Autopopulate a SharePoint Form from URL
November 10, 2010 update: I’ve posted the sequel to this post: Autopopulate a SharePoint Form from URL (with SPUtility.js). I highly recommend checking out the new and improved way using SPUtility.js!
This post builds off of my previous post about making fields read only. In this case, I needed to autopopulate the NewForm.aspx with a value that gets passed to the form via URL parameters.
Editing in SharePoint's SPGridView
The goal of this post is to provide an easy to understand example of how to enable editing in an SPGridView. In this case, my datasource of choice is a SharePoint list.
The way that I chose to implement editing in an SPGridView involved two classes:
- SimpleLogic – handles query and update operations
- SimpleSPGrid – handles all of the setup for the ObjectDataSource and the SPGridView
The main advantage for splitting the two classes is to separate all of the business logic into one class. Ideally, you should be able to completely change how the SimpleLogic class retrieves/updates data without changing the SimpleSPGrid code.
Turning SharePoint SPAuditEntry Update into Create
If you’ve done any work with SharePoint and SPAuditEntry, you may have noticed the distinct lack of Create event. Instead, two Update events are created which means some extra processing is required.
AllowUnsafeUpdates and ValidateFormDigest
Internal column names: Site Columns vs List Columns
Two identical custom lists with the same columns. The only difference is that one list has columns added using Site Columns and the other list had columns added directly to the list. The same? Not so much…
Updating choice fields via SharePoint Web Services
**May 20, 2010 Update
** This post is about using a C# console app to update choice fields via SharePoint web services. If you are looking for more information on how SharePoint web services can be called via JavaScript, check out my post on building a simple SharePoint AJAX app.
My problem started with a simple change to a site column. I had originally created a Country field with the following values:
- US
- Canada
After a change in requirements, the new values for the field were:
- Canada
- Germany
- Great Britain
- Ireland
- United States
After updating the site column, I then needed to update the hundreds of items in the list to use “United States” instead of “US”. Instead of updating hundreds of items manually, I decided to write a small web services console app. (I was also not able to update it in datasheet… due to content approval maybe???)
However, I quickly ran into problems when attempting to update the multi-select choice field.
SharePoint's SPGridView, filtering, and apostrophes
May 20, 2010 Update
If you are looking for a great example on how to create an SPGridView, check out Erik Burger’s blog:
- Building A SPGridView Control – Part 1: Introducing the SPGridView
- Building A SPGridView Control – Part 2: Filtering
This post is really meant to supplement part 2 of Erik’s guide and address values in the grid that include apostrophes when using filtering.
The SPGridView is one of the most useful SharePoint controls. You can use it to do sorting, grouping, and filtering just like the out of the box List View Web Part does for regular lists and libraries. This makes it relatively easy to create a custom grid with your own data. Unfortunately, filtering with the SPGridView is a little quirky, especially if your data could potentially contain apostrophes.
Filtering a SharePoint list by Workflow Status
The most recent problem I ran into was when I tried to filter out items that had a certain Workflow status in the list. More specifically, I was trying to filter on items that had errored out and had a status of “Error Occurred” in the list.
Creating a SharePoint Alert with different EventTypes
I recently ran into a problem creating alerts in a web part for the current user. After the alert was added, no matter what I set the SPAlert’s EventType to, it always defaulted to All. I finally figured out a way around this problem thanks to a helpful post by erwin at sharepointology.
Boolean values in an Event Handler
While researching a bug with editing in datasheet view and a custom list event handler, I discovered strange behavior with boolean values. Any boolean value you get from the SPItemEventDataCollection depends on where the item was originally edited: either using the regular user interface (EditForm.aspx, NewForm.aspx) or using datasheet view.
Creating a link to the current item in a workflow
When creating a workflow using SharePoint Designer, oftentimes it is useful to be able to create a link to the current list item. Unfortunately, the built in columns are not so helpful.
Selected tab in the Top Link Bar
The Danger in using SPSite.OpenWeb()
Recently I ran into a “feature” of SharePoint’s SPSite.OpenWeb() method (the no argument constructor specifically). If the OpenWeb() method is used with a URL that is not known to exist, it can result in some unexpected behavior.
Make a Text field read only on EditForm.aspx without SPD
November 10, 2010 Update: I’ve posted the sequel to this post: Autopopulate a SharePoint Form from URL (with SPUtility.js). The code below will not work in SharePoint 2013 so you should use SPUtility.js!
After looking at a couple of examples online, most of them require the use of SharePoint Designer to make a field read only (or to hide it). Here is a method to make a field readonly using only Prototype (javascript library) and a Content Editor Web Part
Failed to extract the cab file in the solution.
Enterprise Search and SharePoint
Category: Uncategorized
Hugo Bitbucket Pipeline to Deploy to S3
Here is a working Bitbucket pipeline for deploying a Hugo static site to AWS S3.
Migrating from Wordpress to Hugo
I recently decided to migrate my blog from Wordpress to Hugo. I don’t blog as often these days so trying to keep up with new wordpress versions, securing my wordpress installation against hack attempts, and fighting comment spam was becoming overwhelming.
Powershell profile for Java and Maven
As we gradually replace regular windows command line with powershell, it will be useful to set up a powershell environment for Java / Maven development.
Powershell script to change SSRS Subscription Owner
Use jQuery to call the Microsoft Translator AJAX API
First, make sure you have subscribed to the Microsoft Translator API on Azure Marketplace and registered your application Azure DataMarket. For testing, you can use the free subscription which lets you submit up to 2 million characters a month. Registering your app will create a Client ID and a Client Secret which you need to get authorization.
In order to protect your Client Secret, you’ll need to write server-side code to get your access token. The access code is then passed to the Microsoft Translator API via AJAX.
Convert from HandySafe to Sky Wallet
I wrote a small python script to help convert Handy Safe to Sky Wallet. I definitely learned a lot about python and even though I wasn’t very familiar with the available API I was able to come up with a pretty nifty script.
Using slices in Golang
Adding minified javascript/css files to ClearCase
Top 3 Features missing from SSRS 2008 R2
A list of gripes about features I feel are missing from Microsoft SQL Server Reporting Services 2008 R2.
My experience with a fresh Ubuntu 12.04 install
After my botched Windows 8 install, I decided to go back to Ubuntu. I still would like to figure out a solution with Windows on this laptop so that I could use it for work but right now having linux is better. I decided to document some of the issues I ran into in case I need them later.
Web services proxy: Unable to generate a temporary class
I ran across a strange error while deploying a very simple DLL. It contained only one class; a web services proxy class to the Lists SharePoint web service that I generated using the wsdl tool. Here is the error:
Unable to generate a temporary class (result=1).
error CS2001: Source file ‘C:\WINDOWS\TEMP_zm3ve2y.0.cs’ could not be found
error CS2008: No inputs specified
Commit to SharePoint Overflow!
AutoIt3 script to auto resize windows
One thing that bugs me all the time is when a window doesn’t remember the position that it was resized or moved to. Although it is more of a nuisance than anything else, I decided to write a little AutoIt3 script to automatically resize my Remote Desktop windows to be the certain place and size that I prefer them to be in.