IBM Watson NLC and Conversation services (as well as many other NLU cloud platforms) provide a Swift SDK to use in custom apps to implement intent understanding from natural language utterances.
These SDKs and the corresponding NLU platforms are super powerful. They provide much more than simply intent understanding capability — they also detect entities/slots and provide tools to manage complex, long running conversation dialogs.
However, even for the most basic NLC inference, these SDKs depend on network connectivity, as the NLC model is run in the Cloud.
iOS already provides very efficient Text-To-Speech and Speech-To-Text APIs (SFSpeechRecognizer and AVSpeechSynthesizer) fully capable of working offline on iPhone/iPad devices. By using Core ML models to run NLC and NLU algorithms on the device, we can provide similar functionality without relying on cloud inference.
I thought it would be helpful to implement an open source project to implement basic intent understanding functionalities, directly importing datasets of intents and sample utterances designed with those popular NLU cloud platforms.
Table of contents:
SwiftNLC
SwiftNLC is a Natural Language Classifier based on Core ML / TensorFlow integration, capable of running offline on iOS/watchOS/tvOS devices.
This project is available on GitHub at:
SwiftNLC is composed of different sub-projects:
- Importer: A Swift multi platform console app to import intents and utterances from different formats (could run on macOS and Linux)
- SampleDatasets: A folder for JSON files containing intents definitions and sample utterances
- Embedder: A Swift macOS console app to prepare the word embedding encoding using NSLinguisticTagger. It must run on macOS and not on Linux, as Apple NSLinguisticTagger is not part of the public multi-platform Foundation Library
- ModelNotebook: A folder containing Jupyter Notebooks for implementing the Deep Neural Network Classifier model using Keras/TensorFlow API and exporting it using the Apple CoreMLTools Python library
- Wrapper: A Swift wrapper to the auto-generated Core ML model to simplify access to the Core ML Classifier model and prepare/encode the utterances to use for prediction
- SwiftNLCTestClient: A simple test iOS application to play with the wrapper and the Core ML model
Detailed instructions
Importer — Swift script to import from Watson and others
This sub-project contains Swift code (to be executed on Linux or macOS environments) to import a file containing the dataset used to train the NLC model.
The idea is to provide different importers for different file formats in order to import datasets from existing NLU/NLC/Conversation platforms such as IBM Watson, AWS Alexa/Lex, Google Dialogflow, etc.
The first importer implemented is for the IBM Watson Conversation service. Watson uses a JSON “workspace” file containing intents with several sample utterances, entities (slots), also with sample utterances, as well as a tree for complex dialog management of long-running conversation. This project is solely about NLC, and it only import intents and sample utterances from this file.
Usage example:
Generated dataset.json example:
Sample Datasets
This folder contains sample datasets with intents and sample utterances from different sources:
- Watson/WatsonConversationCarWorkspace.json in the Watson subfolder is an export of the standard sample workspace for demoing IBM Watson Conversation service on the IBM Cloud.
- PharmacyDataset.json (no need to import)
Embedder — Word one-hot encoding implemented in Swift using NSLinguisticTagger
This sub-project contains Swift code (to be executed on a macOS or iOS environments) to import a JSON file containing the dataset to be used for training the NLC model.
This importer uses Apple Foundation NSLinguisticTagger APIs to analyze and tokenize the text in the sample utterances, creating a word embedder. In particular, it outputs a one-hot encoding for stem words, a corpus of documents, and a class of entities to train the data and prepare the TensorFlow model, as well as for inferencing the model with Core ML.
Usage example:
This command produces the following files in the current folder: bagOfWords.json, lemmatizedDataset.json and intents.json
ModelNotebook — Instructions to create TensorFlow model
This is a Python Jupyter Notebook using the Keras API and TensorFlow as a backend to create a simple, fully connected Deep Network Classifier.
If you’re new to the Keras/TensorFlow/Jupyter world, here are step-by-step instructions to create the ML model using Keras/TensorFlow and export it on Core ML using CoreMLConversionTool
- Download and install Anaconda Python:
2. Create the Keras, TensorFlow, Python, and Core ML environment:
This environment is created based on the environment.yml file for installing Python 2.7, TensorFlow 1.1, Keras 2.0.4, CoreMLTools 0.6.3, Pandas and other useful Python packages:
NB NLTK is only needed for the createModelWithNLTKEmbedding Notebook used on initial testing.
The final createModelWithNSLinguisticTaggerEmbedding does not use NLTK, as the word embedding is implemented in Swift on the Embedder module using the NSLinguisticTagger API.
3. Activate the environment (Mac/Linux):
4. Check that your prompt changed to:
5. Launch Jupyter Notebook:
6. Open your browser to:
To create a basic model with Keras/TensorFlow and export it with CoreMLTools, open createModelWithNSLinguisticTaggerEmbedding in your Jupyter browsing session and execute any cells in order to create, save, and export the Keras Model using Core ML exporting tools.
The basic Core ML model will be saved in the current folder.
Keras / TensorFlow model consideration
As discussed above, the Swift Embedder project is generally responsible for massaging the dataset, and responsible in particular for creating the one-hot word encoding for stem words from sample utterances and for creating a corpus of stemmed, encoded documents to be used for training the model.
For this reason, the Python code to create the model is simplified with very little pre-processing, and it quickly starts to create the TensorFlow model used for learning.
This model is a simple, fully connected network that receives as input an array of embedded 0’s and 1’s for each sample utterance. It uses ‘ReLU’ as an activator for the first and an internal hidden layer, and then at the end, as this is a multi-class classifier, it uses ‘softmax’ to emphasize the winner prediction.
The training of the model is even simpler, as per definition the intents/utterances dataset used per input is very limited, and there’s no space at all for creating validation and testing sets.
It basically trains the entire dataset with a sufficient number of epochs to obtain maximum possible accuracy.
Once the Keras/TensorFlow model is trained, this is easily exported to Core ML using the Apple CoreMLTools Python library:
Core ML Swift Wrapper and Word Embedding preparation
Once the exported Core ML model is imported in a client app to predict the intent from a new utterance, it will need to be encoded using the same one-hot embedding logic used for preparing the training dataset.
The wrapper folder contains the following Swift files to implement the one-hot embedding using the NSLinguistic tagger and to easily wrap access to the Core ML model. This simplifies the annoying creation of MLMultiArray parameters.
Lemmatizer.swift
import Foundation
class Lemmatizer {
typealias TaggedToken = (String, String?)
func tag(text: String, scheme: String) -> [TaggedToken] {
let options: NSLinguisticTagger.Options = [.omitWhitespace, .omitPunctuation, .omitOther, .joinNames]
let tagger = NSLinguisticTagger(tagSchemes: NSLinguisticTagger.availableTagSchemes(forLanguage: "en"), options: Int(options.rawValue))
tagger.string = text
var tokens: [TaggedToken] = []
tagger.enumerateTags(in: NSMakeRange(0, text.count), scheme:NSLinguisticTagScheme(rawValue: scheme), options: options) { tag, tokenRange, _, _ in
let token = (text as NSString).substring(with: tokenRange)
tokens.append((token, tag?.rawValue))
}
return tokens
}
func lemmatize(text: String) -> [TaggedToken] {
return tag(text: text.lowercased(), scheme: NSLinguisticTagScheme.lemma.rawValue)
}
}BagOfWords.swift
import Foundation
struct BagOfWords: Codable {
let sortedArrayOfWords: [String]
init(setOfWords: Set<String>) {
sortedArrayOfWords = Array(setOfWords).sorted()
}
internal func binarySearch(_ word: String) -> Int? {
var lowerIndex = 0
var upperIndex = sortedArrayOfWords.count - 1
while true {
let currentIndex = (lowerIndex + upperIndex) / 2
if sortedArrayOfWords[currentIndex] == word {
return currentIndex
}
else if lowerIndex > upperIndex {
return nil
}
else {
if sortedArrayOfWords[currentIndex] > word {
upperIndex = currentIndex - 1
}
else {
lowerIndex = currentIndex + 1
}
}
}
}
func embed(arrayOfWords: [String]) -> [Int] {
var embedding = [Int](repeating: 0, count: sortedArrayOfWords.count)
for word in arrayOfWords {
if let index = binarySearch(word) {
embedding[index] = 1
}
}
return embedding
}
}SwiftNLCModel.swift
import Foundation
import CoreML
class SwiftNLCModel {
lazy var bagOfWords: BagOfWords = {
return try! JSONDecoder().decode(BagOfWords.self, from: Data(contentsOf: Bundle.main.url(forResource:"bagOfWords", withExtension: "json")!))
}()
lazy var intents: [String] = {
return try! JSONDecoder().decode(Array<String>.self, from: Data(contentsOf: Bundle.main.url(forResource:"intents", withExtension: "json")!))
}()
var lemmatizer = Lemmatizer()
func predict(_ utterance: String) -> (String, Float)? {
let lemmas = lemmatizer.lemmatize(text: utterance).compactMap { $0.1 }
let embedding = bagOfWords.embed(arrayOfWords: lemmas)
let model = SwiftNLC()
let size = NSNumber(value: embedding.count)
let mlMultiArrayInput = try! MLMultiArray(shape:[size], dataType:MLMultiArrayDataType.double)
for i in 0..<size.intValue {
mlMultiArrayInput[i] = NSNumber(floatLiteral: Double(embedding[i]))
}
let prediction = try! model.prediction(input: SwiftNLCInput(embeddings: mlMultiArrayInput))
var pos = -1
var max:Float = 0.0
for i in 0..<prediction.entities.count {
let p = prediction.entities[i].floatValue
if p > max {
max = p
pos = i
}
}
return pos >= 0 ? (intents[pos], max) : nil
}
}Sample iOS App
Once imported, the Swift wrappers files above, using this SwiftNLC Core ML model, will be easy to execute, as in following code:
let model = SwiftNLCModel()
@IBAction func go(_ sender: Any) {
if let prediction = model.predict(commandField.text!) {
intentLabel.text = "(prediction.0) ((String(format: "%2.1f", prediction.1 * 100))%)"
}
else {
intentLabel.text = "error"
}
}
More details about how to use Core ML in an iOS app are provided in this tutorial:
Future
As I mentioned, this open source project doesn’t aim to replace much more evolved NLC and NLU platforms such as IBM Watson NLC, Conversation, and others.
At the moment, it only supports basic intent understanding and, importantly, it uses a very simple, fully connected neural network with a limited one-hot word encoding technique.
It works quite well and, of course, it works totally offline!
In the future, I’ll extend this project to support more complex embedding methods, such as Word2Vec. I’ll also adopt a more powerful RNN model to try to detect entities/slots.
Discuss this post on Hacker News
Comments 0 Responses