searching for better code: presented by grant ingersoll, lucidworks
TRANSCRIPT
O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
Searching for Better Code Grant Ingersoll
CTO, Lucidworks @gsingers
3
5
• Languages: Java, JS/HTML/CSS, Python, Clojure
• Tooling: Gradle, JIRA, Github, Jenkins, Docker, Slack
• Tests: TestNG, Karma, Pytest, Manual
• Lots of open source, downstream dependencies
• Agile (mostly)
• ~40 people, globally distributed
Lucidworks Eng
Fusion: we wanted to call it Solr++, but our attorney said no!
Connector Framework
Index Pipelines (ETL)
( )ScaleFault ToleranceReal-Time
Fusion APIs
Recommendations Personalization Contextual SearchRelevancy Tool
Machine Learning / Signal ProcessingAnalytics
Security
Apps Mobile Silk
Database Web File Logs Hadoop
Make your data smarter. Ask more interesting questions.
Massive scale. Real-time. Secure.
Any data. Any source.
Connector Framework
Index Pipelines (ETL)
( )ScaleFault ToleranceReal-Time
Fusion APIs
Recommendations Personalization Contextual SearchRelevancy Tool
Machine Learning / Signal ProcessingAnalytics
Security
Apps Mobile Silk
Database Web File Logs Hadoop
Fusion: Solr and moreFeature Solr Fusion
Core Search/Faceting/…Alerting
Connectors JDBC only (60+)Integrated ETL Limited
Built-in RecommenderEditorial Controls Limited
Integrated NLP and Models Limited: UIMAAdmin UI
Fault tolerance/High ScaleSecurity LimitedAnalytics
Signal Capture and feedbackDashboards (reporting, monitoring)
Big data Integrations Limited: HDFS Index StorageLog Search
Billions of Docs
Optional
REST
Security woven throughout
Prox
y/LB
Recs
Worker
Pipes Metrics
NLP Sched.
Blobs Msging
Connectors
Worker Cluster Mgr.
Spark
Shards Shards
Solr
HD
FS
Shared Config Mgmt
Leader Election
Load Balancing
ZK 1
Zookeeper
ZK N
Signals
Fusion Architecture
Millions of Users
11
• Code search
• Operational problems
• Download Stats Analysis
• Heartbeat
Use Cases: Basic Search
Demo I
13
• Connectors
• e.g. JIRA, Github/SVN, Logstash, Zendesk (2.2)
• Pipelines for ETL, per doc analysis
• Spark/Aggregations for large scale analysis
Fusion for Code
14
• Alerting
• PagerDuty, Slack, email
• Visualization: Silk built in!
• Security
• Kerberos, LDAP/AD, custom
Fusion for Code
15
• Code Review
• Static Analysis
• Headers, required configuration
• Team/Developer/Commit/Bug Metrics
• Trends and Hotspots
• Legal issues
Use Cases: Better Code
Demo II
17
• OpenGrok Analysis coming soon (help wanted!)
• http://github.com/gsingers/open-grok-analyzers
• Findbugs, Linters
• IT Operations
• Realtime + Static
• Sales/Support Operations: Zendesk, Apache Mailing Lists, JIRA
Next Steps
18
• Download Fusion 2.1: http://lucidworks.com/products/fusion/
• Check out http://github.com/LucidWorks/fusion-examples
• Contact Me:
• @gsingers on Twitter
More Info
