These are the news items I've curated in my monitoring of the API space that have some relevance to the rogue API conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is testing their APIs, going beyond just monitoring and understand the details of each request and response.15 Nov 2017
<p</p>The concept of rogue APIs isn’t anything new. Instagram started out as a rogue API, and many leading platforms who are less than open with their platforms have rogue APIs. They are usually APIs that have been reverse engineered from mobile applications, and published to Github for other developers to use. I’m looking to marry this concept with my low hanging fruit API work, where I help organizes start their API journey using data and content that is already on their website. Meaning, if it is already available on the web as table, form, or as CSV, spreadsheet, or other machine readable fie, it should be available via an API. As APIs are just the next step in the evolution, this is the logical place for the API journey to begin for many companies, organizations, institutions, and government agencies.
I’ve spidered the entire web site of organizations to extract lists of data sources they should be turning into APIs. I’ve done this at the request of the website owner, as well as without the permission. Honestly, it provides a pretty compelling look at the digital presence for an organization when you harvest raw data like this and publish to a Github repository. It isn’t a view that every organization is ready for, or has thought about. Making it an even more important place for organizations to start their API journey. APIs aren’t just about providing access to your data and content for your partners and 3rd party developers, it is about getting a handle on your digital assets, and how you present and provide access to this digital representation of your organization–something many suck at profoundly.
I’d like to invest more cycles into my low hanging fruit API research. I’d love to take some government agencies and not just identify the low hanging fruit, but actually deploy a rogue API portal, and hang some of the APIs there. I’d like to do this to a couple of companies, institutions, as well as government agencies. I know that I’d get in trouble doing this with some companies, and even other entities, but I think it is a good way to instigate the API conversation, and I am willing to take the chance. Ihad the University of Oklahoma contact me after I scraped their web site, and I think I could recreate the effect with other groups. The trick is doing it in a transparent and observable way, with everything on Github, and communicated in a clear way. So, that someone knows who is behind it, and can reach out to do things in a more formal way–moving from a rogue API, to an official API.
To move this forward I am going to target a single government agency, scrape their website, and any other open daa I can find, and then public an official rogue API portal, and begin hanging some of the APIs there. I’m even going to open up read and write capabilities via the API for any developer who wants to register, and pay for access to the API. I’ll make sure things are clearly marked as being a unofficial rogue API, and provide contact information for anyone looking to communicate with me. I see low hanging fruit rogue APIs as being a way I can get the attention of companies, organizations, institutions, and government agencies when it comes to APIs. Even begin to build awarness and critical mass within a community around the digital assets shared on the website, and now via an API portal. A kind of activist API deployment, and beginning the public API journey.
This goes well beyond the concept of scraping for me. Which I’ve seen a number of startups come and go trying to accomplish. This is about helping show organizations the importance having a website as well as APIs to help counter scraping efforts, and get a better handle on their digital presence. It is meant to start the conversation with some very entrenched folks around the digital resources they are making public, and how APIs can help them better quantify their digital presence, and take control over that presence beyond just their website. If there is an agency, institution, or organization you’d like to see target, or even would be willing to invest some money in deploying a low hanging fruit rogue API portal for, feel free to let me know. I’ll be investing some cycles into this area of my research, just to make sure my content is fresh, while also seeing what new conversations I might be able jumpstart, so its a good time to get involved and fund what I’m doing.
I was looking at how many of the top mobile applications in the iTunes story actually had a public API presence, and was finding it very telling what came up in the Google search results for each company when I searched [company name] + API. It tells a lot about how a company sees the world, when they don’t have a public API presence, but they have a very public mobile application that uses APIs.
An example of this is with Tinder, where the top listings are all Github rogue API repositories, when you Google “Tinder API”. Tinder doesn’t own the conversation when it comes to their own APIs. While the Tinder APIs are public, and well documented, Tinder prefers acting like they are private–they aren’t. Pinterest uses SSL pinning, but there is even a good amount of information out there at how to get around that, making the mapping out and documenting of Tinder APIs a pretty doable thing.
Honestly, I don’t care about Tinder’s APIs. They are just an easy example to point a finger at and use as a poster child. I don’t even expect them to have fully public APIs that any developer could use without permission. Sure, lock that shit down, but provide a sandbox, and make sure every application gets approval before they can more access to live data. Make sure that you own the API conversation by having a developers portal, and provide information regarding what it takes to get access, and maybe some day actually become an approved partner.
I’m not saying that every company should have freely available public APIs. I’m saying every company should own the public conversation around their APIs, no matter what their strategy for developing applications around a platform’s APIs. Have a presence. Own the conversation. Have a door for application developers to walk, even if there is a waiting room. Not all applications will be competing with your own web, mobile, device, or network applications. Some will be about enabling data portability for you users, or maybe provide useful access aggregate data for use in visualizations–you never know what folks will be bringing to the table, why keep the door closed?
I understand. You may not be all team API like I am, but you are using APIs to drive your mobile experience. I just don’t get why you wouldn’t want to own the conversation around these APIs. You are leaving so much on the table. If your mobile app is finding success, people will want access to the goodness going on behind it–a rogue API is what kickstarted the Instagram API in the early days. It is pretty easy to reverse engineer any mobile application, and map out the surface area of the API behind, as well as the authentication in play. Either you own the conversation around your API, or someone will step up and do it for you in todays online world.
I am finally getting the time to invest more into the rest of my API industry guides, which involves deep dives into core areas of my research like API definitions, design, and now deployment. The outline for my API deployment research has begun to come into focus and looks like it will rival my API management research in size.
With this release, I am looking to help onboard some of my less technical readers with API deployment. Not the technical details, but the big picture, so I wanted to start with some simple questions, to help prime the discussion around API development.
- Where? - Where are APIs being deployed. On-premise, and in the clouds. Traditional website hosting, and even containerized and serverless API deployment.
- How? - What technologies are being used to deploy APIs? From using spreadsheets, document and file stores, or the central database. Also thinking smaller with microservices, containes, and serverless.
- Who? - Who will be doing the deployment? Of course, IT and developers groups will be leading the charge, but increasingly business users are leveraging new solutions to play a significant role in how APIs are deployed.
The Role Of API Definitions While not every deployment will be auto-generated using an API definition like OpenAPI, API definitions are increasingly playing a lead role as the contract that doesn’t just deploy an API, but sets the stage for API documentation, testing, monitoring, and a number of other stops along the API lifecycle. I want to make sure to point out in my API deployment research that API definitions aren’t just overlapping with deploying APIs, they are essential to connect API deployments with the rest of the API lifecycle.
Using Open Source Frameworks Early on in this research guide I am focusing on the most common way for developers to deploy an API, using an open source API framework. This is how I deploy my APIs, and there are an increasing number of open source API frameworks available out there, in a variety of programming languages. In this round I am taking the time to highlight at least six separate frameworks in the top programming languages where I am seeing sustained deployment of APIs using a framework. I don’t take a stance on any single API framework, but I do keep an eye on which ones are still active, and enjoying usag bey developers.
Deployment In The Cloud After frameworks, I am making sure to highlight some of the leading approaches to deploying APIs in the cloud, going beyond just a server and framework, and leveraging the next generation of API deployment service providers. I want to make sure that both developers and business users know that there are a growing number of service providers who are willing to assist with deployment, and with some of them, no coding is even necessary. While I still like hand-rolling my APIs using my peferred framework, when it comes to some simpler, more utility APIs, I prefer offloading the heavy lifting to a cloud service, and save me the time getting my hands dirty.
Essential Ingredients for Deployment Whether in the cloud, on-premise, or even on device and even the network, there are some essential ingredients to deploying APIs. In my API deployment guide I wanted to make sure and spend some time focusing on the essential ingredients every API provider will have to think about.
-Compute - The base ingredient for any API, providing the compute under the hood. Whether its baremetal, cloud instances, or serverless, you will need a consistent compute strategy to deploy APIs at any scale. -Storage - Next, I want to make sure my readers are thinking about a comprehensive storage strategy that spans all API operations, and hopefully multiple locations and providers. -DNS - Then I spend some time focusing on the frontline of API deployment–DNS. In todays online environment DNS is more than just addressing for APIs, it is also security. -Encryption - I also make sure encryption is baked in to all API deployment by default in both transit, and storage.
Some Of The Motivations Behind Deploying APIs In previous API deployment guides I usually just listed the services, tools, and other resources I had been aggregating as part of my monitoring of the API space. Slowly I have begun to organize these into a variety of buckets that help speak to many of the motivations I encounter when it comes to deploying APIs. While not a perfect way to look at API deployment, it helps me thinking about the many reasons people are deploying APIs, and craft a narrative, and provide a guide for others to follow, that is potentially aligned with their own motivations.
- Geographic - Thinking about the increasing pressure to deploy APIs in specific geographic regions, leveraging the expansion of the leading cloud providers.
- Virtualization - Considering the fact that not all APIs are meant for production and there is a lot to be learned when it comes to mocking and virtualizing APIs.
- Data - Looking at the simplest of Create, Read, Update, and Delete (CRUD) APIs, and how data is being made more accessible by deploying APIs.
- Database - Also looking at how APIs are beign deployed from relational, noSQL, and other data sources–providing the most common way for APIs to be deployed.
- Spreadsheet - I wanted to make sure and not overlook the ability to deploy APIs directly from a spreadsheet making APIs are within reach of business users.
- Search - Looking at how document and content stores are being indexed and made searchable, browsable, and accessible using APIs.
- Scraping - Another often overlooked way of deploying an API, from the scraped content of other sites–an approach that is alive and well.
- Proxy - Evolving beyond early gateways, using a proxy is still a valid way to deploy an API from existing services.
- Rogue - I also wanted to think more about some of the rogue API deployments I’ve seen out there, where passionate developers reverse engineer mobile apps to deploy a rogue API.
- Microservices - Microservices has provided an interesting motivation for deploying APIs–one that potentially can provide small, very useful and focused API deployments.
- Containers - One of the evolutions in compute that has helped drive the microservices conversation is the containerization of everything, something that compliments the world of APis very well.
- Serverless - Augmenting the microservices and container conversation, serverless is motivating many to think differently about how APIs are being deployed.
- Real Time - Thinking briefly about real time approaches to APIs, something I will be expanding on in future releases, and thinking more about HTTP/2 and evented approaches to API deployment.
- Devices - Considering how APis are beign deployed on device, when it comes to Internet of Things, industrial deployments, as well as even at the network level.
- Marketplaces - Thinking about the role API marketplaces like Mashape (now RapidAPI) play in the decision to deploy APIs, and how other cloud providers like AWS, Google, and Azure will play in this discussion.
- Webhooks - Thinking of API deployment as a two way street. Adding webhooks into the discussion and making sure we are thinking about how webhooks can alleviate the load on APIs, and push data and content to external locations.
- Orchestration - Considering the impact of continous integration and deployment on API deploy specifically, and looking at it through the lens of the API lifecycle.
I feel like API deployment is still all over the place. The mandate for API management was much better articulated by API service providers like Mashery, 3Scale, and Apigee. Nobody has taken the lead when it came to API deployment. Service providers like DreamFactory and Restlet have kicked ass when it comes to not just API management, but making sure API deployment was also part of the puzzle. Newer API service providers like Tyk are also pusing the envelope, but I still don’t have the number of API deployment providers I’d like, when it comes to referring my readers. It isn’t a coincidence that DreamFactory, Restlet, and Tyk are API Evangelist partners, it is because they have the services I want to be able to recommend to my readers.
This is the first time I have felt like my API deployment research has been in any sort of focus. I carved this layer of my research of my API management research some years ago, but I really couldn’t articulate it very well beyond just open source frameworks, and the emerging cloud service providers. After I publish this edition of my API deployment guide I’m going to spend some time in the 17 areas of my research listed above. All these areas are heavily focused on API deployment, but I also think they are all worth looking at individually, so that I can better understand where they also intersect with other areas like management, testing, monitoring, security, and other stops along the API lifecycle.
I am coming across more API providers who have carved off specific "skills" derived from their API, and offering up as part of the latest push to acquire new users on Slack or Facebook. Services like Github, Heroku, and Runscope that API providers and developers are putting to work increasingly have bots they employ, extending their API driven solutions to Slack and Facebook.
Alongside having an application gallery, and having an iPaaS solution showcase, maybe it's time to start having a dedicated page to showcase the bot solutions that are built on your API. Of course, these would start with your own bot solutions, but like application galleries, you could have bots that were built within your community as well.
I'm not going to add a dedicated bot showcase page until I've seen at least a handful in the wild, but I like documenting these things as I think of them. It gives me some dates to better understand at which point did certain things in the API universe begin expanding (or not). Also if you are doing a lot of bot development around your API, or maybe your community is, it might be the little nudge you need to be one of the first APIs out there with a dedicated bot showcase page.
I see quite a few rogue APIs, and often rogue SDKs, but this is the first time I've come across a rogue embeddable button. While browsing Product Hunt this morning I came across this rogue Snapchat embeddable button, which allows you to promote your Snapchat account on any website.
It just makes me sad that platforms don't just ignore platform fans and advocates like this, but actively work to lock things down to prevent this kind of serendipity from happening. Why would you want to shut down people who are looking to promote your tool? You should be enabling them.
If Snapchat had a proper API portal, they would take this signal, internalize it, and turn it into an official set of embeddable tools for the messaging platform. #derp
I'm going to keep beating the patent API drumbeat, until I bring more awareness to the topic, and shine a light on what is going on. While I will still be my usual self and call out the worst behavior in the space, I am also going to try and be a little more friendlier around my views, and try and help bring more options to the table. This is a serious problem, nobody is talking about, and one that has many dimensions and nuances--if you want my raw stance on API patents, you can read it here.
One area I wanted to try and cover, in response to my friends trying to convince me their aren't bad people, in having patents. I know you aren't, and it isn't my goal to make you look bad in this, it is only to shine light on the entire process, how broken it is, and call out the worst offenders. If you truly believe in patents, protecting the work you've done, and that your intentions are good, share your patent portfolio with the world, and showcase it like you do the other aspects of the work you do. You will craft a press release about everything else you do, do the same for your patents.
I do not think patents are bad. I think ill-conceived patent ideas, that haven't been properly vetted by the under resourced USPTO, that are used in back door dealings as leverage, and litigated in a court of law are bad. I'll take your word that your patents are good, and you aren't operating in any of these areas, if you are public, transparent, and openly proud of them, as you state in private conversations.
Part of the purpose of my research is to encourage good behavior in the sector, by highlight the common building blocks of the space. I think I will add a patent portfolio building to my research. While I have ZERO examples to highlight, I encourage API companies to do this, and would love to highlight in a positive way, any company that is straight up enough to showcase their patents. If you are proud of your API patents, and do not have bad intentions in having them, please publish your portfolio, show case them as you would anything else you are doing--help bring API patents out of the shadows.
Time Tracking API platform Harvest has embraced Github as part of their API ecosystem. I'm always on the hunt for examples of API providers using Github, so I figured I'd showcase Harvest's creative use of the social coding platform.
Starting with their documentation, the Harvest team has moved the API documentation to a Github repository, allowing developers to "watch" the API, get updates when changes are made, asks questions or even contribute to the API docs by submitting a pull request.
Harvest is also using the wiki portion of their Github repo for a developer application gallery they are calling Community Creations and Hacks, where they showcase innovative uses of the Harvest API--currently displaying 20 integrations by Harvest users.
I'm currently tracking on 11 separate uses of Github for API management, and always on the hunt for new ways to use Github to support API ecosystems. Nice move Harvest!
I stumbled across the Twitter Counter API in my monitoring for the API Stack this morning. The Twitter Counter API allows you to retrieve key metrics on any Twitter account like username, url and avatar. All data you can get via the Twitter API, but with Twitter Counter API you get additional information like account growth statistics and ranking, that Twitter doesn't provide at all.
I find it fascinating that someone can build an API to augment an existing API, which is why I keep talking about it, I guess :) We are seeing a more standardized version of this with API aggregation providers like Singly and Adigami, where they not only aggregate APIs from a variety of sources, they also build entirely new APIs based the added value that is created after they are brought together.
Thinking about if further, it would be cool if you could submit your API to be listed in your parent API providers API area. Think of APIhub and Mashape, but every API area would have its own 3rd API marketplace. API providers often allow 3rd party developers to submit code libraries and samples to be listed as resources, as well as applications for listing in an application showcase. So it makes sense to potetially allow for your developers to submit APIs for validation and publishing into a designated area.
Poster boy for how to properly run your API ecosystem properly, Twilio, recently updated their DOer Gallery to highlight developers in the Twilio ecosystem that build cool stuff on the popular voice and SMS API.
Twilio has the best record I’ve seen of any API, when it comes to showcasing and being loved by their developer community, and I'm sure the DOer Gallery plays an important role in that.
The Twilio DOer Gallery has the following features:
- Personal Details
- Short Bio
- Other Profiles
Devloper Galleries like Twilios might not be for every API platform. But if you have a passionate base of developers you might want to consider giving them their own profile and a gallery where they can not just discover and interact with each other, it can let other companies find potential developers to execute projects via your API.
A Developer Gallery can be a great way to give your API developers some love and attention. Twilio even features developers from their DOer Gallery on their blog in a "DOer of the Month".
Would showcasing your “API DOers” benefit your API community?
Factual has launch a new application gallery to showcase the diverse number of applications built using data provided by Factual.
You can search for apps, browse by category, and filter by open source, paid or free apps. Looks like there are about 18 apps in the directory currently ranging from augmented reality to daily deals.
The Factual App Gallery isn’t a particularly unique launch, we are seeing app showcases popup within many APIs, but it shows that Factual is gaining steam, and I think it shows the appetite for building apps around datasets is growing.
- 1st prize - Makerbot Thing-O-Matic 3D Printer
- 2nd prize - 1TB USB hard drive enclosed in a vintage nintendo game (Zelda, Metroid, etc)
- 3rd prize - Set of BuckyBalls magnetic building spheres
- NewIn - This application shows new members joining LinkedIn from around the world.
- ChromeIn - Integrate LinkedIn directly into Google Chrome. Easy access to your LinkedIn updates, anytime.
- Signal - Signal is aimed at making it easy for all professionals to glean the most relevant insights from the never-ending stream of status updates and news. An API Labs is a great way to showcase experimental and innovative projects that utilize your API.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.